[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260620 19:56:51.272209 5074 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.244.190:36613
I20260620 19:56:51.273872 5074 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:56:51.275022 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:56:51.293957 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:56:51.294013 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:56:51.303534 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:56:51.295034 5074 server_base.cc:1061] running on GCE node
I20260620 19:56:52.492800 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:56:52.493038 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:56:52.493281 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985412493260 us; error 0 us; skew 500 ppm
I20260620 19:56:52.494169 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:56:52.501151 5074 webserver.cc:533] Webserver started at http://127.4.244.190:37649/ using document root <none> and password file <none>
I20260620 19:56:52.502019 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:56:52.502254 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:56:52.502779 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:56:52.508850 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/master-0-root/instance:
uuid: "288e961cf54049eba9a0909a850e73dc"
format_stamp: "Formatted at 2026-06-20 19:56:52 on dist-test-slave-zxgc"
I20260620 19:56:52.516561 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20260620 19:56:52.521907 5089 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:52.523206 5074 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 19:56:52.523576 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/master-0-root
uuid: "288e961cf54049eba9a0909a850e73dc"
format_stamp: "Formatted at 2026-06-20 19:56:52 on dist-test-slave-zxgc"
I20260620 19:56:52.523924 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:56:52.580852 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:56:52.582782 5074 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:56:52.583247 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:56:52.662248 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.190:36613
I20260620 19:56:52.662323 5140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.190:36613 every 8 connection(s)
I20260620 19:56:52.670866 5141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:52.693482 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc: Bootstrap starting.
I20260620 19:56:52.699664 5141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:52.701877 5141 log.cc:826] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:56:52.706404 5141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc: No bootstrap required, opened a new log
I20260620 19:56:52.728124 5141 raft_consensus.cc:359] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e961cf54049eba9a0909a850e73dc" member_type: VOTER }
I20260620 19:56:52.728672 5141 raft_consensus.cc:385] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:52.728974 5141 raft_consensus.cc:740] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288e961cf54049eba9a0909a850e73dc, State: Initialized, Role: FOLLOWER
I20260620 19:56:52.729921 5141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e961cf54049eba9a0909a850e73dc" member_type: VOTER }
I20260620 19:56:52.730417 5141 raft_consensus.cc:399] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:56:52.730753 5141 raft_consensus.cc:493] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:56:52.731139 5141 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:52.735230 5141 raft_consensus.cc:515] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e961cf54049eba9a0909a850e73dc" member_type: VOTER }
I20260620 19:56:52.736403 5141 leader_election.cc:304] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 288e961cf54049eba9a0909a850e73dc; no voters:
I20260620 19:56:52.737962 5141 leader_election.cc:290] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:56:52.738356 5144 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:52.741106 5144 raft_consensus.cc:697] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 1 LEADER]: Becoming Leader. State: Replica: 288e961cf54049eba9a0909a850e73dc, State: Running, Role: LEADER
I20260620 19:56:52.742108 5144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e961cf54049eba9a0909a850e73dc" member_type: VOTER }
I20260620 19:56:52.743376 5141 sys_catalog.cc:565] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:56:52.753698 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "288e961cf54049eba9a0909a850e73dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e961cf54049eba9a0909a850e73dc" member_type: VOTER } }
I20260620 19:56:52.753883 5145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 288e961cf54049eba9a0909a850e73dc. Latest consensus state: current_term: 1 leader_uuid: "288e961cf54049eba9a0909a850e73dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "288e961cf54049eba9a0909a850e73dc" member_type: VOTER } }
I20260620 19:56:52.754418 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [sys.catalog]: This master's current role is: LEADER
I20260620 19:56:52.754421 5145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [sys.catalog]: This master's current role is: LEADER
I20260620 19:56:52.758675 5153 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 19:56:52.770116 5153 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 19:56:52.772328 5074 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260620 19:56:52.788301 5153 catalog_manager.cc:1382] Generated new cluster ID: 259dcfcf6b2c4fe18e92a3bf043d2e25
I20260620 19:56:52.788610 5153 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 19:56:52.818707 5153 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 19:56:52.820284 5153 catalog_manager.cc:1539] Loading token signing keys...
I20260620 19:56:52.835405 5153 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc: Generated new TSK 0
I20260620 19:56:52.836534 5153 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 19:56:52.903738 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:56:52.913014 5163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:56:52.915398 5164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:56:52.919071 5074 server_base.cc:1061] running on GCE node
W20260620 19:56:52.919862 5166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:56:52.920809 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:56:52.921043 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:56:52.921232 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985412921212 us; error 0 us; skew 500 ppm
I20260620 19:56:52.921828 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:56:52.932539 5074 webserver.cc:533] Webserver started at http://127.4.244.129:34151/ using document root <none> and password file <none>
I20260620 19:56:52.933471 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:56:52.933797 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:56:52.934197 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:56:52.935961 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-0-root/instance:
uuid: "e2c0f888513f4dbca42e2df83cee8a02"
format_stamp: "Formatted at 2026-06-20 19:56:52 on dist-test-slave-zxgc"
I20260620 19:56:52.943733 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 19:56:52.949980 5171 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:52.950963 5074 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 19:56:52.951399 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-0-root
uuid: "e2c0f888513f4dbca42e2df83cee8a02"
format_stamp: "Formatted at 2026-06-20 19:56:52 on dist-test-slave-zxgc"
I20260620 19:56:52.951882 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:56:52.985468 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:56:52.987819 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:56:52.990823 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:56:52.996655 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:56:52.997001 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:52.997416 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:56:52.997738 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:53.105269 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.129:34041
I20260620 19:56:53.105401 5233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.129:34041 every 8 connection(s)
I20260620 19:56:53.113665 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:56:53.126374 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:56:53.135458 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:56:53.148468 5074 server_base.cc:1061] running on GCE node
W20260620 19:56:53.149446 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:56:53.150585 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:56:53.150940 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:56:53.151218 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985413151196 us; error 0 us; skew 500 ppm
I20260620 19:56:53.152043 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:56:53.155012 5234 heartbeater.cc:344] Connected to a master server at 127.4.244.190:36613
I20260620 19:56:53.155503 5234 heartbeater.cc:461] Registering TS with master...
I20260620 19:56:53.156869 5234 heartbeater.cc:507] Master 127.4.244.190:36613 requested a full tablet report, sending...
I20260620 19:56:53.158697 5074 webserver.cc:533] Webserver started at http://127.4.244.130:43309/ using document root <none> and password file <none>
I20260620 19:56:53.159387 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:56:53.159643 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:56:53.160004 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:56:53.160652 5106 ts_manager.cc:194] Registered new tserver with Master: e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041)
I20260620 19:56:53.161629 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-1-root/instance:
uuid: "8a5e3300aab24352be03d75663278916"
format_stamp: "Formatted at 2026-06-20 19:56:53 on dist-test-slave-zxgc"
I20260620 19:56:53.164232 5106 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55022
I20260620 19:56:53.169305 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260620 19:56:53.173768 5246 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:53.175069 5074 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:56:53.175396 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-1-root
uuid: "8a5e3300aab24352be03d75663278916"
format_stamp: "Formatted at 2026-06-20 19:56:53 on dist-test-slave-zxgc"
I20260620 19:56:53.175704 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:56:53.201392 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:56:53.202744 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:56:53.205169 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:56:53.207533 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:56:53.207779 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:53.208015 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:56:53.208185 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:53.310468 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.130:38111
I20260620 19:56:53.310667 5308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.130:38111 every 8 connection(s)
I20260620 19:56:53.317831 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:56:53.326159 5312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:56:53.328060 5313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:56:53.336401 5309 heartbeater.cc:344] Connected to a master server at 127.4.244.190:36613
I20260620 19:56:53.336987 5309 heartbeater.cc:461] Registering TS with master...
I20260620 19:56:53.338111 5309 heartbeater.cc:507] Master 127.4.244.190:36613 requested a full tablet report, sending...
I20260620 19:56:53.341189 5106 ts_manager.cc:194] Registered new tserver with Master: 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:53.343449 5106 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55026
W20260620 19:56:53.349623 5315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:56:53.350417 5074 server_base.cc:1061] running on GCE node
I20260620 19:56:53.351416 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:56:53.351663 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:56:53.351836 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985413351821 us; error 0 us; skew 500 ppm
I20260620 19:56:53.352381 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:56:53.354949 5074 webserver.cc:533] Webserver started at http://127.4.244.131:36217/ using document root <none> and password file <none>
I20260620 19:56:53.355429 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:56:53.355639 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:56:53.355902 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:56:53.356905 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-2-root/instance:
uuid: "9f431a428f5c4feb98f2c984e13311a7"
format_stamp: "Formatted at 2026-06-20 19:56:53 on dist-test-slave-zxgc"
I20260620 19:56:53.361618 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 19:56:53.364959 5320 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:53.365765 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:56:53.366039 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-2-root
uuid: "9f431a428f5c4feb98f2c984e13311a7"
format_stamp: "Formatted at 2026-06-20 19:56:53 on dist-test-slave-zxgc"
I20260620 19:56:53.366307 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1781985411095902-5074-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:56:53.398380 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:56:53.399634 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:56:53.401736 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:56:53.404297 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:56:53.404489 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:53.404788 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:56:53.404939 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:56:53.511598 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.131:35885
I20260620 19:56:53.511724 5382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.131:35885 every 8 connection(s)
I20260620 19:56:53.529642 5383 heartbeater.cc:344] Connected to a master server at 127.4.244.190:36613
I20260620 19:56:53.529997 5383 heartbeater.cc:461] Registering TS with master...
I20260620 19:56:53.530826 5383 heartbeater.cc:507] Master 127.4.244.190:36613 requested a full tablet report, sending...
I20260620 19:56:53.532830 5106 ts_manager.cc:194] Registered new tserver with Master: 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:53.532939 5074 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015949252s
I20260620 19:56:53.534317 5106 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55032
I20260620 19:56:53.588624 5106 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55034:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:56:53.592324 5106 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:56:53.713621 5348 tablet_service.cc:1511] Processing CreateTablet for tablet fb69c1c8316344a2bc9557bb2ce94238 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260620 19:56:53.715190 5346 tablet_service.cc:1511] Processing CreateTablet for tablet c9a086e8da4c43a0b3ef18242b74bf2f (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260620 19:56:53.716143 5345 tablet_service.cc:1511] Processing CreateTablet for tablet 77e89736f48a46978fd607ea294db738 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260620 19:56:53.715386 5347 tablet_service.cc:1511] Processing CreateTablet for tablet 4d070e48b2424cc480e243cce4e7fc53 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260620 19:56:53.717847 5346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a086e8da4c43a0b3ef18242b74bf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.721716 5348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb69c1c8316344a2bc9557bb2ce94238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.728026 5199 tablet_service.cc:1511] Processing CreateTablet for tablet fb69c1c8316344a2bc9557bb2ce94238 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260620 19:56:53.729326 5199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb69c1c8316344a2bc9557bb2ce94238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.732290 5198 tablet_service.cc:1511] Processing CreateTablet for tablet 4d070e48b2424cc480e243cce4e7fc53 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260620 19:56:53.733634 5198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d070e48b2424cc480e243cce4e7fc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.738111 5347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d070e48b2424cc480e243cce4e7fc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.739874 5197 tablet_service.cc:1511] Processing CreateTablet for tablet c9a086e8da4c43a0b3ef18242b74bf2f (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260620 19:56:53.735566 5271 tablet_service.cc:1511] Processing CreateTablet for tablet 77e89736f48a46978fd607ea294db738 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260620 19:56:53.750279 5197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a086e8da4c43a0b3ef18242b74bf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.752163 5345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77e89736f48a46978fd607ea294db738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.749022 5272 tablet_service.cc:1511] Processing CreateTablet for tablet c9a086e8da4c43a0b3ef18242b74bf2f (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260620 19:56:53.754210 5272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a086e8da4c43a0b3ef18242b74bf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.749787 5270 tablet_service.cc:1511] Processing CreateTablet for tablet 0fa02cada6aa4a179ef372d4ce3a0fc9 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260620 19:56:53.746100 5274 tablet_service.cc:1511] Processing CreateTablet for tablet fb69c1c8316344a2bc9557bb2ce94238 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260620 19:56:53.757947 5344 tablet_service.cc:1511] Processing CreateTablet for tablet 0fa02cada6aa4a179ef372d4ce3a0fc9 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260620 19:56:53.759872 5271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77e89736f48a46978fd607ea294db738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.762980 5270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa02cada6aa4a179ef372d4ce3a0fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.747709 5273 tablet_service.cc:1511] Processing CreateTablet for tablet 4d070e48b2424cc480e243cce4e7fc53 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260620 19:56:53.757388 5196 tablet_service.cc:1511] Processing CreateTablet for tablet 77e89736f48a46978fd607ea294db738 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
W20260620 19:56:53.766160 5154 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:56:53.769321 5344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa02cada6aa4a179ef372d4ce3a0fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.755365 5195 tablet_service.cc:1511] Processing CreateTablet for tablet 0fa02cada6aa4a179ef372d4ce3a0fc9 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260620 19:56:53.772615 5274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb69c1c8316344a2bc9557bb2ce94238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.775635 5196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77e89736f48a46978fd607ea294db738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.778254 5194 tablet_service.cc:1511] Processing CreateTablet for tablet 6a5d1fb6c9b0439d99aa9c8447b33ed3 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260620 19:56:53.781229 5273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d070e48b2424cc480e243cce4e7fc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.791484 5194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5d1fb6c9b0439d99aa9c8447b33ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.811175 5401 tablet_bootstrap.cc:492] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:53.813355 5195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa02cada6aa4a179ef372d4ce3a0fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.842789 5269 tablet_service.cc:1511] Processing CreateTablet for tablet 6a5d1fb6c9b0439d99aa9c8447b33ed3 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260620 19:56:53.846904 5269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5d1fb6c9b0439d99aa9c8447b33ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.890880 5193 tablet_service.cc:1511] Processing CreateTablet for tablet 7066f7b456e4438e93cd393e270a554b (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260620 19:56:53.892328 5401 tablet_bootstrap.cc:654] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:53.892997 5193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7066f7b456e4438e93cd393e270a554b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.897248 5268 tablet_service.cc:1511] Processing CreateTablet for tablet 7066f7b456e4438e93cd393e270a554b (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260620 19:56:53.899870 5268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7066f7b456e4438e93cd393e270a554b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.901957 5346 tablet_service.cc:1511] Processing CreateTablet for tablet 7066f7b456e4438e93cd393e270a554b (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260620 19:56:53.903242 5346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7066f7b456e4438e93cd393e270a554b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.919787 5192 tablet_service.cc:1511] Processing CreateTablet for tablet 2c3ac57b2aec4191ae5f5f818c07c1ce (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260620 19:56:53.923625 5192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c3ac57b2aec4191ae5f5f818c07c1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.920441 5345 tablet_service.cc:1511] Processing CreateTablet for tablet 6a5d1fb6c9b0439d99aa9c8447b33ed3 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260620 19:56:53.932818 5267 tablet_service.cc:1511] Processing CreateTablet for tablet 2c3ac57b2aec4191ae5f5f818c07c1ce (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260620 19:56:53.949726 5348 tablet_service.cc:1511] Processing CreateTablet for tablet 2c3ac57b2aec4191ae5f5f818c07c1ce (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260620 19:56:53.955418 5267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c3ac57b2aec4191ae5f5f818c07c1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.960268 5191 tablet_service.cc:1511] Processing CreateTablet for tablet f1ba73a8009d466f9db25d83eec1af67 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260620 19:56:53.962023 5191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ba73a8009d466f9db25d83eec1af67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.959399 5345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a5d1fb6c9b0439d99aa9c8447b33ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.973214 5266 tablet_service.cc:1511] Processing CreateTablet for tablet f1ba73a8009d466f9db25d83eec1af67 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260620 19:56:53.978016 5266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ba73a8009d466f9db25d83eec1af67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:53.989773 5343 tablet_service.cc:1511] Processing CreateTablet for tablet f1ba73a8009d466f9db25d83eec1af67 (DEFAULT_TABLE table=test-workload [id=384934e3317847bc8eeb33466e0dcfb8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260620 19:56:54.011608 5348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c3ac57b2aec4191ae5f5f818c07c1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:54.023487 5402 tablet_bootstrap.cc:492] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.031490 5343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ba73a8009d466f9db25d83eec1af67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:56:54.034162 5401 tablet_bootstrap.cc:492] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.034821 5401 ts_tablet_manager.cc:1403] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.224s user 0.012s sys 0.017s
I20260620 19:56:54.039914 5401 raft_consensus.cc:359] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.040691 5401 raft_consensus.cc:385] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.040985 5401 raft_consensus.cc:740] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.042388 5401 consensus_queue.cc:260] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.080935 5404 tablet_bootstrap.cc:492] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.082295 5383 heartbeater.cc:499] Master 127.4.244.190:36613 was elected leader, sending a full tablet report...
I20260620 19:56:54.082437 5401 ts_tablet_manager.cc:1434] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.047s user 0.014s sys 0.008s
I20260620 19:56:54.083428 5401 tablet_bootstrap.cc:492] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:54.093325 5402 tablet_bootstrap.cc:654] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.098500 5404 tablet_bootstrap.cc:654] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.104209 5401 tablet_bootstrap.cc:654] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.104513 5405 raft_consensus.cc:493] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.105716 5405 raft_consensus.cc:515] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.112618 5405 leader_election.cc:290] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.162461 5401 tablet_bootstrap.cc:492] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.163650 5401 ts_tablet_manager.cc:1403] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.019s
I20260620 19:56:54.167824 5402 tablet_bootstrap.cc:492] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.168294 5402 ts_tablet_manager.cc:1403] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.146s user 0.018s sys 0.009s
I20260620 19:56:54.170260 5234 heartbeater.cc:499] Master 127.4.244.190:36613 was elected leader, sending a full tablet report...
I20260620 19:56:54.172556 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.175166 5401 raft_consensus.cc:359] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.176169 5401 raft_consensus.cc:385] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.172675 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.178476 5401 raft_consensus.cc:740] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
W20260620 19:56:54.180341 5323 leader_election.cc:343] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:56:54.181663 5321 leader_election.cc:343] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.181463 5404 tablet_bootstrap.cc:492] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.182071 5321 leader_election.cc:304] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7; no voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:54.180646 5401 consensus_queue.cc:260] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.182476 5404 ts_tablet_manager.cc:1403] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.014s
I20260620 19:56:54.183427 5405 raft_consensus.cc:2749] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:56:54.180614 5402 raft_consensus.cc:359] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.184700 5402 raft_consensus.cc:385] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.184986 5402 raft_consensus.cc:740] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.185657 5402 consensus_queue.cc:260] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.186403 5404 raft_consensus.cc:359] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.192394 5402 ts_tablet_manager.cc:1434] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.024s user 0.007s sys 0.004s
I20260620 19:56:54.193327 5402 tablet_bootstrap.cc:492] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.193486 5401 ts_tablet_manager.cc:1434] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260620 19:56:54.194438 5401 tablet_bootstrap.cc:492] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:54.188982 5404 raft_consensus.cc:385] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.197959 5404 raft_consensus.cc:740] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.199604 5404 consensus_queue.cc:260] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.204561 5401 tablet_bootstrap.cc:654] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.206089 5402 tablet_bootstrap.cc:654] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.208221 5309 heartbeater.cc:499] Master 127.4.244.190:36613 was elected leader, sending a full tablet report...
I20260620 19:56:54.209306 5404 ts_tablet_manager.cc:1434] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.026s user 0.004s sys 0.009s
I20260620 19:56:54.210343 5404 tablet_bootstrap.cc:492] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.218250 5404 tablet_bootstrap.cc:654] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.221200 5402 tablet_bootstrap.cc:492] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.221724 5402 ts_tablet_manager.cc:1403] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.004s
I20260620 19:56:54.224435 5402 raft_consensus.cc:359] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.225180 5402 raft_consensus.cc:385] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.225399 5401 tablet_bootstrap.cc:492] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.225504 5402 raft_consensus.cc:740] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.226106 5401 ts_tablet_manager.cc:1403] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.007s
I20260620 19:56:54.226526 5402 consensus_queue.cc:260] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.230098 5401 raft_consensus.cc:359] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.231257 5401 raft_consensus.cc:385] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.231565 5401 raft_consensus.cc:740] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.232306 5401 consensus_queue.cc:260] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.233865 5404 tablet_bootstrap.cc:492] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.234704 5404 ts_tablet_manager.cc:1403] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.000s
I20260620 19:56:54.235481 5401 ts_tablet_manager.cc:1434] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20260620 19:56:54.236809 5401 tablet_bootstrap.cc:492] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:54.240595 5402 ts_tablet_manager.cc:1434] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260620 19:56:54.243978 5401 tablet_bootstrap.cc:654] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.244037 5404 raft_consensus.cc:359] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.244859 5404 raft_consensus.cc:385] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.245203 5404 raft_consensus.cc:740] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.245929 5404 consensus_queue.cc:260] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.248056 5402 tablet_bootstrap.cc:492] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.249102 5404 ts_tablet_manager.cc:1434] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.014s user 0.001s sys 0.005s
I20260620 19:56:54.250205 5404 tablet_bootstrap.cc:492] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.255839 5401 tablet_bootstrap.cc:492] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.256273 5401 ts_tablet_manager.cc:1403] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260620 19:56:54.262912 5401 raft_consensus.cc:359] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.263638 5401 raft_consensus.cc:385] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.263913 5401 raft_consensus.cc:740] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.264592 5401 consensus_queue.cc:260] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.265300 5404 tablet_bootstrap.cc:654] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.267342 5401 ts_tablet_manager.cc:1434] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260620 19:56:54.268276 5401 tablet_bootstrap.cc:492] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
W20260620 19:56:54.268270 5384 tablet.cc:2406] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:56:54.274688 5402 tablet_bootstrap.cc:654] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.275154 5401 tablet_bootstrap.cc:654] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.283259 5404 tablet_bootstrap.cc:492] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.283704 5404 ts_tablet_manager.cc:1403] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.006s
I20260620 19:56:54.284272 5401 tablet_bootstrap.cc:492] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.284636 5401 ts_tablet_manager.cc:1403] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20260620 19:56:54.286262 5404 raft_consensus.cc:359] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.286458 5401 raft_consensus.cc:359] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.287477 5404 raft_consensus.cc:385] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.287544 5401 raft_consensus.cc:385] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.287828 5404 raft_consensus.cc:740] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.287910 5401 raft_consensus.cc:740] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.288707 5401 consensus_queue.cc:260] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.290494 5402 tablet_bootstrap.cc:492] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.290928 5402 ts_tablet_manager.cc:1403] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.007s
I20260620 19:56:54.291131 5401 ts_tablet_manager.cc:1434] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:56:54.292097 5401 tablet_bootstrap.cc:492] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:54.293911 5402 raft_consensus.cc:359] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.294585 5402 raft_consensus.cc:385] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.294871 5402 raft_consensus.cc:740] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.288782 5404 consensus_queue.cc:260] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.297183 5404 ts_tablet_manager.cc:1434] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260620 19:56:54.297689 5402 consensus_queue.cc:260] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.299618 5401 tablet_bootstrap.cc:654] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.299765 5402 ts_tablet_manager.cc:1434] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 19:56:54.300698 5402 tablet_bootstrap.cc:492] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.303256 5404 tablet_bootstrap.cc:492] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.313263 5402 tablet_bootstrap.cc:654] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.317761 5404 tablet_bootstrap.cc:654] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.333323 5402 tablet_bootstrap.cc:492] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.333290 5412 raft_consensus.cc:493] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.334061 5402 ts_tablet_manager.cc:1403] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.006s
I20260620 19:56:54.333933 5412 raft_consensus.cc:515] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.337409 5412 leader_election.cc:290] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.338846 5401 tablet_bootstrap.cc:492] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.339402 5401 ts_tablet_manager.cc:1403] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.048s user 0.024s sys 0.016s
I20260620 19:56:54.343607 5401 raft_consensus.cc:359] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.344311 5401 raft_consensus.cc:385] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.344616 5401 raft_consensus.cc:740] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.345955 5402 raft_consensus.cc:359] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.347668 5402 raft_consensus.cc:385] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.347132 5401 consensus_queue.cc:260] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.348737 5404 tablet_bootstrap.cc:492] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.349332 5404 ts_tablet_manager.cc:1403] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.008s
I20260620 19:56:54.350780 5401 ts_tablet_manager.cc:1434] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260620 19:56:54.351749 5401 tablet_bootstrap.cc:492] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:54.355487 5404 raft_consensus.cc:359] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.356832 5404 raft_consensus.cc:385] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.357152 5404 raft_consensus.cc:740] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.348565 5402 raft_consensus.cc:740] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.365780 5401 tablet_bootstrap.cc:654] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.363085 5404 consensus_queue.cc:260] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.363407 5402 consensus_queue.cc:260] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.369854 5404 ts_tablet_manager.cc:1434] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.020s user 0.010s sys 0.001s
I20260620 19:56:54.377178 5404 tablet_bootstrap.cc:492] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.372107 5402 ts_tablet_manager.cc:1434] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20260620 19:56:54.382863 5358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:54.383618 5358 raft_consensus.cc:2468] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 0.
I20260620 19:56:54.385134 5402 tablet_bootstrap.cc:492] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.385494 5173 leader_election.cc:304] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.387104 5412 raft_consensus.cc:2804] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:54.387458 5412 raft_consensus.cc:493] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:54.387801 5412 raft_consensus.cc:3060] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.391273 5411 raft_consensus.cc:493] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.391752 5411 raft_consensus.cc:515] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.392524 5401 tablet_bootstrap.cc:492] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.392997 5401 ts_tablet_manager.cc:1403] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.009s
I20260620 19:56:54.394124 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.394829 5284 raft_consensus.cc:2468] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 0.
I20260620 19:56:54.396301 5411 leader_election.cc:290] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.395304 5412 raft_consensus.cc:515] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.397229 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.397294 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.397938 5284 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 0.
I20260620 19:56:54.399890 5321 leader_election.cc:304] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:56:54.400647 5411 raft_consensus.cc:2804] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:54.400997 5411 raft_consensus.cc:493] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:54.401559 5358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:54.401321 5411 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.402293 5358 raft_consensus.cc:3060] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.400331 5412 leader_election.cc:290] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.403523 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:54.404136 5284 raft_consensus.cc:3060] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.404912 5209 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 0.
I20260620 19:56:54.400352 5401 raft_consensus.cc:359] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.407970 5401 raft_consensus.cc:385] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.408423 5401 raft_consensus.cc:740] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.409227 5401 consensus_queue.cc:260] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.410797 5413 raft_consensus.cc:493] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.412736 5413 raft_consensus.cc:515] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.416157 5413 leader_election.cc:290] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.418079 5404 tablet_bootstrap.cc:654] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.416932 5358 raft_consensus.cc:2468] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 1.
I20260620 19:56:54.419113 5402 tablet_bootstrap.cc:654] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.424628 5173 leader_election.cc:304] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.425335 5401 ts_tablet_manager.cc:1434] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
I20260620 19:56:54.425671 5412 raft_consensus.cc:2804] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:54.412264 5411 raft_consensus.cc:515] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.429742 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:54.430876 5209 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.432307 5401 tablet_bootstrap.cc:492] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:54.437256 5411 leader_election.cc:290] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.436647 5284 raft_consensus.cc:2468] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 1.
I20260620 19:56:54.441423 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:54.445104 5283 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.461184 5411 raft_consensus.cc:493] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.461792 5411 raft_consensus.cc:515] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.466580 5412 raft_consensus.cc:697] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:54.470820 5411 leader_election.cc:290] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.475843 5404 tablet_bootstrap.cc:492] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.476461 5404 ts_tablet_manager.cc:1403] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.104s user 0.023s sys 0.012s
I20260620 19:56:54.476648 5283 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 1.
I20260620 19:56:54.477077 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.478210 5321 leader_election.cc:304] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:56:54.478492 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.472841 5209 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 1.
I20260620 19:56:54.479604 5411 raft_consensus.cc:2804] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 19:56:54.480265 5321 leader_election.cc:343] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:56:54.482558 5323 leader_election.cc:343] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.471231 5412 consensus_queue.cc:237] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.482349 5411 raft_consensus.cc:697] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Becoming Leader. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:56:54.484668 5432 raft_consensus.cc:493] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.487653 5402 tablet_bootstrap.cc:492] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.486233 5411 consensus_queue.cc:237] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.488744 5402 ts_tablet_manager.cc:1403] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.008s
I20260620 19:56:54.488364 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.492039 5404 raft_consensus.cc:359] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.493245 5404 raft_consensus.cc:385] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.493611 5404 raft_consensus.cc:740] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
W20260620 19:56:54.492161 5249 leader_election.cc:343] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.494515 5404 consensus_queue.cc:260] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.490415 5432 raft_consensus.cc:515] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.502315 5358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:54.502923 5358 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 0.
I20260620 19:56:54.503221 5401 tablet_bootstrap.cc:654] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.502199 5404 ts_tablet_manager.cc:1434] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.025s user 0.005s sys 0.006s
I20260620 19:56:54.505785 5248 leader_election.cc:304] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters: e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:54.505636 5402 raft_consensus.cc:359] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.507645 5404 tablet_bootstrap.cc:492] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.507663 5413 raft_consensus.cc:2804] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:54.507670 5402 raft_consensus.cc:385] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.508949 5413 raft_consensus.cc:493] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:54.509827 5413 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.509117 5402 raft_consensus.cc:740] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.515928 5402 consensus_queue.cc:260] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.517979 5432 leader_election.cc:290] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.520129 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.520195 5358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:54.520831 5284 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 0.
I20260620 19:56:54.521059 5358 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 0.
I20260620 19:56:54.531752 5172 leader_election.cc:304] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.555258 5431 raft_consensus.cc:2804] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:54.532676 5413 raft_consensus.cc:515] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.557063 5431 raft_consensus.cc:493] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:54.560238 5431 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.566221 5413 leader_election.cc:290] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.566442 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:54.566833 5358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:54.573763 5358 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:56:54.577724 5249 leader_election.cc:343] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.584860 5401 tablet_bootstrap.cc:492] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.585278 5401 ts_tablet_manager.cc:1403] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.154s user 0.012s sys 0.015s
I20260620 19:56:54.585894 5323 leader_election.cc:304] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7; no voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:54.588573 5411 raft_consensus.cc:2749] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 19:56:54.588699 5323 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:34041, user_credentials={real_user=slave}} blocked reactor thread for 107783us
I20260620 19:56:54.597518 5358 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 1.
I20260620 19:56:54.600713 5402 ts_tablet_manager.cc:1434] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.111s user 0.010s sys 0.025s
I20260620 19:56:54.600859 5248 leader_election.cc:304] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters: e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:54.602284 5402 tablet_bootstrap.cc:492] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.602500 5413 raft_consensus.cc:2804] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:54.599272 5431 raft_consensus.cc:515] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:56:54.605195 5401 raft_consensus.cc:359] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.605933 5401 raft_consensus.cc:385] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.606241 5401 raft_consensus.cc:740] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.606487 5431 leader_election.cc:290] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Requested vote from peers 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.607398 5401 consensus_queue.cc:260] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.607738 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:54.608307 5284 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.609589 5401 ts_tablet_manager.cc:1434] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.024s user 0.014s sys 0.005s
I20260620 19:56:54.610405 5401 tablet_bootstrap.cc:492] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7: Bootstrap starting.
I20260620 19:56:54.611254 5404 tablet_bootstrap.cc:654] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.615787 5284 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 1.
I20260620 19:56:54.618745 5172 leader_election.cc:304] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.619834 5431 raft_consensus.cc:2804] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:54.620589 5431 raft_consensus.cc:697] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:54.621641 5431 consensus_queue.cc:237] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
W20260620 19:56:54.625197 5173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:35885, user_credentials={real_user=slave}} blocked reactor thread for 93976.2us
I20260620 19:56:54.623808 5413 raft_consensus.cc:697] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:54.627074 5358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:54.627820 5358 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.628947 5438 raft_consensus.cc:493] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.627930 5413 consensus_queue.cc:237] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.629789 5401 tablet_bootstrap.cc:654] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.629385 5438 raft_consensus.cc:515] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.631340 5411 raft_consensus.cc:493] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.631898 5411 raft_consensus.cc:515] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.636480 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.635247 5102 catalog_manager.cc:5696] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "9f431a428f5c4feb98f2c984e13311a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260620 19:56:54.640898 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.638429 5209 raft_consensus.cc:2468] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 0.
I20260620 19:56:54.648473 5323 leader_election.cc:304] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.634195 5105 catalog_manager.cc:5696] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:54.653858 5405 raft_consensus.cc:2804] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:54.654502 5405 raft_consensus.cc:493] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:54.654974 5405 raft_consensus.cc:3060] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.658918 5411 leader_election.cc:290] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.667811 5358 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 1.
I20260620 19:56:54.659931 5438 leader_election.cc:290] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.671252 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:54.672770 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.677623 5284 raft_consensus.cc:2468] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 0.
I20260620 19:56:54.678185 5411 raft_consensus.cc:493] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.678676 5411 raft_consensus.cc:515] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.679332 5405 raft_consensus.cc:515] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.680150 5402 tablet_bootstrap.cc:654] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.680600 5357 raft_consensus.cc:2393] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a5e3300aab24352be03d75663278916 in current term 1: Already voted for candidate 9f431a428f5c4feb98f2c984e13311a7 in this term.
I20260620 19:56:54.682169 5405 leader_election.cc:290] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.684248 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.685115 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:54.686779 5208 raft_consensus.cc:3060] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.686861 5411 leader_election.cc:290] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
W20260620 19:56:54.686239 5323 leader_election.cc:343] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.689685 5437 raft_consensus.cc:493] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.690311 5437 raft_consensus.cc:515] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.691313 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:54.692612 5284 raft_consensus.cc:3060] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.693993 5404 tablet_bootstrap.cc:492] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.694589 5404 ts_tablet_manager.cc:1403] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.187s user 0.022s sys 0.059s
I20260620 19:56:54.695546 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
W20260620 19:56:54.696959 5321 leader_election.cc:343] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.697378 5321 leader_election.cc:304] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7; no voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:54.698102 5411 raft_consensus.cc:2749] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:56:54.698979 5404 raft_consensus.cc:359] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.701756 5437 leader_election.cc:290] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.702315 5404 raft_consensus.cc:385] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.701879 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.702725 5404 raft_consensus.cc:740] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.703588 5404 consensus_queue.cc:260] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.704322 5208 raft_consensus.cc:2468] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 1.
I20260620 19:56:54.704435 5358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:54.705169 5358 raft_consensus.cc:2468] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 0.
I20260620 19:56:54.705420 5323 leader_election.cc:304] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.706184 5404 ts_tablet_manager.cc:1434] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20260620 19:56:54.706280 5411 raft_consensus.cc:2804] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:54.706768 5411 raft_consensus.cc:697] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Becoming Leader. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:56:54.707325 5404 tablet_bootstrap.cc:492] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.707541 5411 consensus_queue.cc:237] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.708808 5173 leader_election.cc:304] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.709754 5437 raft_consensus.cc:2804] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:54.710093 5437 raft_consensus.cc:493] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:54.710402 5437 raft_consensus.cc:3060] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:56:54.710484 5172 leader_election.cc:343] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.711195 5248 leader_election.cc:304] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a5e3300aab24352be03d75663278916; no voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:54.711807 5284 raft_consensus.cc:2468] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 1.
I20260620 19:56:54.713169 5413 raft_consensus.cc:2749] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:56:54.720616 5401 tablet_bootstrap.cc:492] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7: No bootstrap required, opened a new log
I20260620 19:56:54.721028 5401 ts_tablet_manager.cc:1403] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7: Time spent bootstrapping tablet: real 0.111s user 0.013s sys 0.025s
I20260620 19:56:54.725169 5437 raft_consensus.cc:515] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.726732 5401 raft_consensus.cc:359] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.727538 5401 raft_consensus.cc:385] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.727852 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:54.727969 5401 raft_consensus.cc:740] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.727705 5106 catalog_manager.cc:5696] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
W20260620 19:56:54.729054 5172 leader_election.cc:343] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.729103 5401 consensus_queue.cc:260] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.732016 5404 tablet_bootstrap.cc:654] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.732401 5401 ts_tablet_manager.cc:1434] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260620 19:56:54.740234 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:54.740849 5357 raft_consensus.cc:3060] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.742506 5402 tablet_bootstrap.cc:492] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.743005 5402 ts_tablet_manager.cc:1403] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.141s user 0.021s sys 0.005s
I20260620 19:56:54.748018 5402 raft_consensus.cc:359] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.748585 5357 raft_consensus.cc:2468] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 1.
I20260620 19:56:54.748698 5402 raft_consensus.cc:385] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.749733 5173 leader_election.cc:304] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters: 8a5e3300aab24352be03d75663278916
I20260620 19:56:54.750152 5402 raft_consensus.cc:740] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.750509 5437 leader_election.cc:290] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.750550 5431 raft_consensus.cc:2804] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:54.751000 5402 consensus_queue.cc:260] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.751255 5431 raft_consensus.cc:697] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:54.752326 5431 consensus_queue.cc:237] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.753958 5404 tablet_bootstrap.cc:492] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.754392 5404 ts_tablet_manager.cc:1403] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.016s
I20260620 19:56:54.757203 5404 raft_consensus.cc:359] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.757995 5404 raft_consensus.cc:385] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.758296 5404 raft_consensus.cc:740] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.759491 5402 ts_tablet_manager.cc:1434] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20260620 19:56:54.759079 5404 consensus_queue.cc:260] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.761744 5404 ts_tablet_manager.cc:1434] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260620 19:56:54.762749 5404 tablet_bootstrap.cc:492] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.764515 5402 tablet_bootstrap.cc:492] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.771399 5404 tablet_bootstrap.cc:654] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.773548 5402 tablet_bootstrap.cc:654] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.782618 5105 catalog_manager.cc:5696] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:54.785404 5105 catalog_manager.cc:5696] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
W20260620 19:56:54.788072 5154 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:56:54.791086 5404 tablet_bootstrap.cc:492] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
I20260620 19:56:54.791570 5404 ts_tablet_manager.cc:1403] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.003s
I20260620 19:56:54.794065 5404 raft_consensus.cc:359] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.794766 5404 raft_consensus.cc:385] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.795078 5404 raft_consensus.cc:740] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.795745 5404 consensus_queue.cc:260] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.797621 5404 ts_tablet_manager.cc:1434] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260620 19:56:54.798419 5404 tablet_bootstrap.cc:492] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916: Bootstrap starting.
I20260620 19:56:54.803669 5404 tablet_bootstrap.cc:654] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.808892 5402 tablet_bootstrap.cc:492] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.809348 5402 ts_tablet_manager.cc:1403] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.024s
I20260620 19:56:54.809486 5106 catalog_manager.cc:5696] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "9f431a428f5c4feb98f2c984e13311a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260620 19:56:54.812927 5402 raft_consensus.cc:359] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.813635 5402 raft_consensus.cc:385] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.813915 5402 raft_consensus.cc:740] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.814630 5402 consensus_queue.cc:260] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.816761 5402 ts_tablet_manager.cc:1434] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:56:54.817631 5402 tablet_bootstrap.cc:492] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.822376 5411 raft_consensus.cc:493] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.822844 5411 raft_consensus.cc:515] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.825187 5411 leader_election.cc:290] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:54.826164 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.826488 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:54.827478 5402 tablet_bootstrap.cc:654] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
W20260620 19:56:54.827498 5321 leader_election.cc:343] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.825709 5404 tablet_bootstrap.cc:492] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916: No bootstrap required, opened a new log
W20260620 19:56:54.828148 5323 leader_election.cc:343] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:56:54.828627 5404 ts_tablet_manager.cc:1403] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.001s
I20260620 19:56:54.828536 5323 leader_election.cc:304] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7; no voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:54.829259 5411 raft_consensus.cc:2749] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:56:54.831308 5404 raft_consensus.cc:359] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.831995 5404 raft_consensus.cc:385] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.832314 5404 raft_consensus.cc:740] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.833065 5404 consensus_queue.cc:260] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.834371 5431 raft_consensus.cc:493] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:54.835153 5404 ts_tablet_manager.cc:1434] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:56:54.834815 5431 raft_consensus.cc:515] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.837087 5402 tablet_bootstrap.cc:492] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.837541 5402 ts_tablet_manager.cc:1403] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20260620 19:56:54.838101 5431 leader_election.cc:290] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.840018 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:54.840611 5357 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 0.
I20260620 19:56:54.840890 5402 raft_consensus.cc:359] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.841542 5402 raft_consensus.cc:385] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.841784 5402 raft_consensus.cc:740] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.842065 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:54.842365 5402 consensus_queue.cc:260] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.842667 5284 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 0.
I20260620 19:56:54.844226 5402 ts_tablet_manager.cc:1434] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:56:54.844278 5173 leader_election.cc:304] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.845180 5437 raft_consensus.cc:2804] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:54.845108 5402 tablet_bootstrap.cc:492] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02: Bootstrap starting.
I20260620 19:56:54.845575 5437 raft_consensus.cc:493] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:54.845925 5437 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.850533 5402 tablet_bootstrap.cc:654] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02: Neither blocks nor log segments found. Creating new log.
I20260620 19:56:54.864622 5437 raft_consensus.cc:515] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.868330 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:54.868327 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:54.869017 5284 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.869063 5357 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:54.870939 5402 tablet_bootstrap.cc:492] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02: No bootstrap required, opened a new log
I20260620 19:56:54.871376 5402 ts_tablet_manager.cc:1403] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.012s
I20260620 19:56:54.875459 5437 leader_election.cc:290] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:54.876921 5284 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 1.
I20260620 19:56:54.877175 5357 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 1.
I20260620 19:56:54.877725 5402 raft_consensus.cc:359] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.878294 5173 leader_election.cc:304] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:54.878530 5402 raft_consensus.cc:385] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:56:54.879246 5437 raft_consensus.cc:2804] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:54.879477 5402 raft_consensus.cc:740] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Initialized, Role: FOLLOWER
I20260620 19:56:54.880210 5402 consensus_queue.cc:260] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.879762 5437 raft_consensus.cc:697] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:54.881717 5402 ts_tablet_manager.cc:1434] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260620 19:56:54.881417 5437 consensus_queue.cc:237] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:54.891053 5102 catalog_manager.cc:5696] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:54.935668 5411 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:54.953475 5437 consensus_queue.cc:1048] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:54.957705 5411 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:54.969187 5412 consensus_queue.cc:1048] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.031963 5413 raft_consensus.cc:493] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:55.032554 5413 raft_consensus.cc:515] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.034297 5413 leader_election.cc:290] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.035125 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:55.035694 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:55.036545 5437 raft_consensus.cc:493] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:55.036975 5437 raft_consensus.cc:515] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.037518 5357 raft_consensus.cc:2393] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a5e3300aab24352be03d75663278916 in current term 1: Already voted for candidate e2c0f888513f4dbca42e2df83cee8a02 in this term.
I20260620 19:56:55.037801 5209 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.038793 5248 leader_election.cc:304] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a5e3300aab24352be03d75663278916; no voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:55.039779 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a5e3300aab24352be03d75663278916" is_pre_election: true
I20260620 19:56:55.039951 5413 raft_consensus.cc:3060] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.042532 5437 leader_election.cc:290] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.043756 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:55.044653 5357 raft_consensus.cc:2393] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2c0f888513f4dbca42e2df83cee8a02 in current term 1: Already voted for candidate 8a5e3300aab24352be03d75663278916 in this term.
I20260620 19:56:55.046036 5173 leader_election.cc:304] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2c0f888513f4dbca42e2df83cee8a02; no voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7
I20260620 19:56:55.046308 5438 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.047101 5437 raft_consensus.cc:2749] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:56:55.047133 5413 raft_consensus.cc:2749] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:56:55.062338 5413 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:56:55.101662 5468 raft_consensus.cc:493] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:55.102671 5468 raft_consensus.cc:515] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.104226 5468 leader_election.cc:290] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.104928 5468 raft_consensus.cc:493] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:56:55.105161 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:55.105295 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:55.105404 5468 raft_consensus.cc:515] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.105803 5208 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 0.
I20260620 19:56:55.105957 5357 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 0.
I20260620 19:56:55.107079 5249 leader_election.cc:304] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:55.107637 5468 leader_election.cc:290] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.108107 5468 raft_consensus.cc:2804] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:55.108433 5468 raft_consensus.cc:493] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:55.108767 5468 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.108868 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7" is_pre_election: true
I20260620 19:56:55.108958 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02" is_pre_election: true
I20260620 19:56:55.109668 5208 raft_consensus.cc:2468] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 0.
I20260620 19:56:55.110088 5357 raft_consensus.cc:2468] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 0.
I20260620 19:56:55.110693 5249 leader_election.cc:304] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:55.111661 5413 raft_consensus.cc:2804] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:56:55.112020 5413 raft_consensus.cc:493] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:56:55.112340 5413 raft_consensus.cc:3060] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.115892 5468 raft_consensus.cc:515] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.117503 5468 leader_election.cc:290] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.118185 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:55.118511 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:55.118775 5208 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.119284 5357 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.119119 5413 raft_consensus.cc:515] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.122133 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:55.122694 5209 raft_consensus.cc:3060] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.126519 5357 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 1.
I20260620 19:56:55.127688 5248 leader_election.cc:304] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:56:55.128415 5468 raft_consensus.cc:2804] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:55.128865 5468 raft_consensus.cc:697] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:55.128753 5209 raft_consensus.cc:2468] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 1.
I20260620 19:56:55.129745 5468 consensus_queue.cc:237] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.130084 5249 leader_election.cc:304] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:55.130626 5413 leader_election.cc:290] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 1 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.130194 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:55.131233 5438 raft_consensus.cc:2804] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:56:55.131700 5438 raft_consensus.cc:697] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:55.132283 5357 raft_consensus.cc:3060] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:56:55.132491 5438 consensus_queue.cc:237] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.134573 5208 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 1.
I20260620 19:56:55.140247 5357 raft_consensus.cc:2468] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 1.
I20260620 19:56:55.148150 5102 catalog_manager.cc:5696] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:55.149523 5102 catalog_manager.cc:5696] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:55.152185 5437 consensus_queue.cc:1048] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.158232 5437 consensus_queue.cc:1048] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.180850 5431 consensus_queue.cc:1048] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.189798 5431 consensus_queue.cc:1048] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.268996 5461 consensus_queue.cc:1048] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.269687 5411 consensus_queue.cc:1048] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.319398 5431 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.331074 5431 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.512930 5357 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
mode: GRACEFUL
new_leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:35798
I20260620 19:56:55.513515 5357 raft_consensus.cc:606] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Received request to transfer leadership to TS e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:55.526742 5283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:37128
I20260620 19:56:55.527237 5283 raft_consensus.cc:606] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Received request to transfer leadership to TS e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:56:55.543224 5208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:57838
I20260620 19:56:55.543784 5208 raft_consensus.cc:606] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:56:55.545179 5438 consensus_queue.cc:1048] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.551148 5356 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:35798
I20260620 19:56:55.551700 5356 raft_consensus.cc:606] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:56:55.557531 5208 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:57838
I20260620 19:56:55.558039 5208 raft_consensus.cc:606] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:56:55.563216 5469 consensus_queue.cc:1048] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.585315 5411 raft_consensus.cc:993] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: : Instructing follower e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:56:55.585785 5411 raft_consensus.cc:1081] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Signalling peer e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:56:55.585706 5283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:37128
I20260620 19:56:55.586411 5283 raft_consensus.cc:606] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Received request to transfer leadership to TS 9f431a428f5c4feb98f2c984e13311a7
I20260620 19:56:55.588862 5208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:57810
I20260620 19:56:55.589504 5208 raft_consensus.cc:493] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:55.590315 5208 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.596180 5208 raft_consensus.cc:515] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.597817 5208 leader_election.cc:290] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 2 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.598563 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:55.598869 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:55.599434 5357 raft_consensus.cc:3055] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:55.599133 5283 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.599720 5357 raft_consensus.cc:740] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:56:55.600526 5357 consensus_queue.cc:260] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.601424 5357 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.606415 5283 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 2.
I20260620 19:56:55.606848 5357 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 2.
I20260620 19:56:55.607690 5172 leader_election.cc:304] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:55.608531 5412 raft_consensus.cc:2804] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:55.608997 5412 raft_consensus.cc:697] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:55.609746 5412 consensus_queue.cc:237] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.616573 5106 catalog_manager.cc:5696] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 1 to 2, leader changed from 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131) to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:55.671176 5468 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.674296 5469 raft_consensus.cc:993] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: : Instructing follower e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:56:55.674782 5469 raft_consensus.cc:1081] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Signalling peer e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:56:55.676685 5208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:57822
I20260620 19:56:55.677452 5208 raft_consensus.cc:493] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:55.677840 5208 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.680918 5469 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:56:55.694316 5208 raft_consensus.cc:515] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.697573 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:55.698144 5283 raft_consensus.cc:3055] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:55.698468 5283 raft_consensus.cc:740] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:55.699386 5283 consensus_queue.cc:260] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.700549 5283 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.703397 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:55.704948 5357 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.707477 5208 leader_election.cc:290] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 2 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.712839 5357 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 2.
I20260620 19:56:55.714285 5173 leader_election.cc:304] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:55.717499 5412 raft_consensus.cc:2804] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:55.718082 5412 raft_consensus.cc:697] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:55.719061 5412 consensus_queue.cc:237] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.722139 5283 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 2.
I20260620 19:56:55.734743 5105 catalog_manager.cc:5696] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 1 to 2, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:55.783901 5405 raft_consensus.cc:993] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:56:55.784389 5461 raft_consensus.cc:1081] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:56:55.785796 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37104
I20260620 19:56:55.786260 5283 raft_consensus.cc:493] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:55.786580 5283 raft_consensus.cc:3060] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.790974 5283 raft_consensus.cc:515] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.792646 5283 leader_election.cc:290] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.793370 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:55.793711 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a5d1fb6c9b0439d99aa9c8447b33ed3" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:55.793969 5209 raft_consensus.cc:3060] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.794309 5357 raft_consensus.cc:3055] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:55.794673 5357 raft_consensus.cc:740] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:56:55.795452 5357 consensus_queue.cc:260] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.796531 5357 raft_consensus.cc:3060] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.803797 5357 raft_consensus.cc:2468] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:56:55.804881 5248 leader_election.cc:304] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:56:55.805698 5469 raft_consensus.cc:2804] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:55.806146 5469 raft_consensus.cc:697] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:55.806460 5209 raft_consensus.cc:2468] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:56:55.806941 5469 consensus_queue.cc:237] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.810462 5432 raft_consensus.cc:993] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:56:55.810915 5432 raft_consensus.cc:1081] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:56:55.812563 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37114
I20260620 19:56:55.813920 5283 raft_consensus.cc:493] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:55.814579 5283 raft_consensus.cc:3060] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.815716 5106 catalog_manager.cc:5696] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 1 to 2, leader changed from 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:55.820008 5283 raft_consensus.cc:515] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.822873 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:55.823446 5209 raft_consensus.cc:3055] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:55.823864 5283 leader_election.cc:290] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:55.823735 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7066f7b456e4438e93cd393e270a554b" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:55.823751 5209 raft_consensus.cc:740] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:55.824536 5357 raft_consensus.cc:3060] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.825707 5209 consensus_queue.cc:260] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.826961 5209 raft_consensus.cc:3060] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:55.830240 5357 raft_consensus.cc:2468] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:56:55.831202 5248 leader_election.cc:304] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:56:55.831857 5468 raft_consensus.cc:2804] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:55.832280 5468 raft_consensus.cc:697] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:55.832973 5468 consensus_queue.cc:237] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:55.833801 5209 raft_consensus.cc:2468] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:56:55.840058 5106 catalog_manager.cc:5696] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 1 to 2, leader changed from e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:55.999251 5357 raft_consensus.cc:1275] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.000458 5432 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.008050 5283 raft_consensus.cc:1275] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.010030 5431 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.028393 5412 raft_consensus.cc:993] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:56:56.028838 5431 raft_consensus.cc:1081] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:56:56.030205 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37114
I20260620 19:56:56.030726 5283 raft_consensus.cc:493] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:56.031028 5283 raft_consensus.cc:3060] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:56.035245 5283 raft_consensus.cc:515] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:56.036736 5283 leader_election.cc:290] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:56:56.037506 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:56.037716 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fa02cada6aa4a179ef372d4ce3a0fc9" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:56:56.038034 5209 raft_consensus.cc:3055] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:56.038201 5357 raft_consensus.cc:3060] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:56.038350 5209 raft_consensus.cc:740] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:56.039054 5209 consensus_queue.cc:260] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:56.040030 5209 raft_consensus.cc:3060] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:56.042999 5357 raft_consensus.cc:2468] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:56:56.044173 5248 leader_election.cc:304] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:56:56.044888 5209 raft_consensus.cc:2468] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:56:56.044955 5468 raft_consensus.cc:2804] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:56.045526 5468 raft_consensus.cc:697] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:56.046320 5468 consensus_queue.cc:237] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:56.052446 5106 catalog_manager.cc:5696] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 1 to 2, leader changed from e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:56:56.126667 5469 raft_consensus.cc:993] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916: : Instructing follower 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:56:56.127135 5468 raft_consensus.cc:1081] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Signalling peer 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:56:56.128571 5357 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:35788
I20260620 19:56:56.129040 5357 raft_consensus.cc:493] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:56.129323 5357 raft_consensus.cc:3060] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:56.134917 5357 raft_consensus.cc:515] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:56.136574 5357 leader_election.cc:290] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 2 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:56.137369 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:56.137598 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1ba73a8009d466f9db25d83eec1af67" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:56.137888 5209 raft_consensus.cc:3060] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:56.138170 5283 raft_consensus.cc:3055] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:56.138470 5283 raft_consensus.cc:740] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:56.139207 5283 consensus_queue.cc:260] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:56.140165 5283 raft_consensus.cc:3060] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:56.142688 5209 raft_consensus.cc:2468] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 2.
I20260620 19:56:56.143635 5323 leader_election.cc:304] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:56.144454 5461 raft_consensus.cc:2804] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:56.144874 5461 raft_consensus.cc:697] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Becoming Leader. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:56:56.145043 5283 raft_consensus.cc:2468] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 2.
I20260620 19:56:56.145649 5461 consensus_queue.cc:237] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:56.151531 5106 catalog_manager.cc:5696] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 reported cstate change: term changed from 1 to 2, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "9f431a428f5c4feb98f2c984e13311a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260620 19:56:56.234730 5357 raft_consensus.cc:1275] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.236130 5468 consensus_queue.cc:1048] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.243746 5209 raft_consensus.cc:1275] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.245210 5469 consensus_queue.cc:1048] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.270783 5283 raft_consensus.cc:1275] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.272040 5412 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.281728 5357 raft_consensus.cc:1275] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.283031 5432 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.294724 5357 raft_consensus.cc:1275] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.295866 5469 consensus_queue.cc:1048] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.303646 5209 raft_consensus.cc:1275] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.305277 5438 consensus_queue.cc:1048] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.470834 5357 raft_consensus.cc:1275] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.472142 5438 consensus_queue.cc:1048] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:56:56.480247 5209 raft_consensus.cc:1275] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.481582 5468 consensus_queue.cc:1048] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:56:56.530299 5209 raft_consensus.cc:1275] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.531548 5461 consensus_queue.cc:1048] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:56.545706 5283 raft_consensus.cc:1275] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:56.547537 5461 consensus_queue.cc:1048] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:58.614768 5074 auto_leader_rebalancer-test.cc:294] The leader distribution is
I20260620 19:56:58.615461 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:56:58.635397 5284 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:54498
I20260620 19:56:58.635895 5284 raft_consensus.cc:606] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Received request to transfer leadership to TS 9f431a428f5c4feb98f2c984e13311a7
I20260620 19:56:58.645669 5209 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77e89736f48a46978fd607ea294db738"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
mode: GRACEFUL
new_leader_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:38900
I20260620 19:56:58.646132 5209 raft_consensus.cc:606] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Received request to transfer leadership to TS 9f431a428f5c4feb98f2c984e13311a7
I20260620 19:56:58.647267 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:56:58.647804 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:56:58.728812 5468 raft_consensus.cc:993] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: : Instructing follower 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:56:58.729298 5499 raft_consensus.cc:1081] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Signalling peer 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:56:58.730784 5356 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:35788
I20260620 19:56:58.731249 5356 raft_consensus.cc:493] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:58.731554 5356 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:58.736253 5356 raft_consensus.cc:515] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:58.737948 5356 leader_election.cc:290] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 2 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:58.738886 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:58.738889 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:58.739650 5284 raft_consensus.cc:3055] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:58.739692 5209 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:58.740111 5284 raft_consensus.cc:740] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:56:58.740945 5284 consensus_queue.cc:260] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:58.742127 5284 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:58.746335 5209 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 2.
I20260620 19:56:58.747699 5323 leader_election.cc:304] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:56:58.748492 5461 raft_consensus.cc:2804] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:58.748569 5284 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 2.
I20260620 19:56:58.750265 5461 raft_consensus.cc:697] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Becoming Leader. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:56:58.751271 5461 consensus_queue.cc:237] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:58.761780 5105 catalog_manager.cc:5696] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 reported cstate change: term changed from 1 to 2, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "9f431a428f5c4feb98f2c984e13311a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260620 19:56:59.028342 5412 raft_consensus.cc:993] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02: : Instructing follower 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:56:59.028749 5509 raft_consensus.cc:1081] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Signalling peer 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:56:59.030131 5356 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "77e89736f48a46978fd607ea294db738"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:35776
I20260620 19:56:59.030547 5356 raft_consensus.cc:493] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:56:59.030817 5356 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:59.035435 5356 raft_consensus.cc:515] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:59.037096 5356 leader_election.cc:290] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 2 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:56:59.037940 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:56:59.038121 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:56:59.038548 5209 raft_consensus.cc:3055] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:56:59.038690 5284 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:59.038997 5209 raft_consensus.cc:740] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:56:59.039860 5209 consensus_queue.cc:260] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:59.041081 5209 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:56:59.044418 5284 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 2.
I20260620 19:56:59.045518 5321 leader_election.cc:304] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:56:59.046278 5461 raft_consensus.cc:2804] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:56:59.046659 5209 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 2.
I20260620 19:56:59.046821 5461 raft_consensus.cc:697] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Becoming Leader. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:56:59.047631 5461 consensus_queue.cc:237] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:56:59.054522 5105 catalog_manager.cc:5696] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 reported cstate change: term changed from 1 to 2, leader changed from e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129) to 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "9f431a428f5c4feb98f2c984e13311a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260620 19:56:59.274339 5209 raft_consensus.cc:1275] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:59.276018 5503 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:59.277587 5284 raft_consensus.cc:1275] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:59.278873 5503 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:59.612308 5209 raft_consensus.cc:1275] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:59.613461 5461 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:59.622257 5284 raft_consensus.cc:1275] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:56:59.623495 5515 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:56:59.648650 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:56:59.653458 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:56:59.653771 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:00.654888 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:00.659797 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:00.660229 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:01.661463 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:01.667034 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:01.667382 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:02.668543 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:02.673251 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:02.673636 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:03.674490 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:03.679270 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:03.679647 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:04.680554 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:04.685256 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:04.685648 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:05.686473 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:05.691016 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:05.691390 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:06.692344 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:06.696919 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:06.697381 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:07.698263 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:07.702844 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:07.703181 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:08.730621 5074 auto_leader_rebalancer-test.cc:313] The leader distribution is
I20260620 19:57:08.735285 5209 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:57838
I20260620 19:57:08.735842 5209 raft_consensus.cc:606] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:57:08.741103 5209 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:57838
I20260620 19:57:08.741490 5209 raft_consensus.cc:606] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:57:08.746311 5209 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:57838
I20260620 19:57:08.746783 5209 raft_consensus.cc:606] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:57:08.751323 5356 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77e89736f48a46978fd607ea294db738"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:35798
I20260620 19:57:08.751694 5356 raft_consensus.cc:606] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:57:08.763078 5356 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
mode: GRACEFUL
new_leader_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:35798
I20260620 19:57:08.763479 5356 raft_consensus.cc:606] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Received request to transfer leadership to TS 8a5e3300aab24352be03d75663278916
I20260620 19:57:08.793196 5412 raft_consensus.cc:993] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:08.793685 5526 raft_consensus.cc:1081] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:08.795089 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37114
I20260620 19:57:08.795588 5283 raft_consensus.cc:493] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:08.795893 5283 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:08.800366 5283 raft_consensus.cc:515] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:08.801968 5283 leader_election.cc:290] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:08.803017 5356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:08.802831 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:57:08.803537 5356 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:08.803597 5209 raft_consensus.cc:3055] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:08.803964 5209 raft_consensus.cc:740] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:57:08.804533 5209 consensus_queue.cc:260] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:08.805411 5209 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:08.808538 5356 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:08.809645 5248 leader_election.cc:304] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:57:08.809993 5412 raft_consensus.cc:993] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:08.810366 5412 raft_consensus.cc:1081] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:08.810472 5523 raft_consensus.cc:2804] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:08.811563 5209 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:08.811655 5523 raft_consensus.cc:697] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:08.812368 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37114
I20260620 19:57:08.813244 5283 raft_consensus.cc:493] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:08.813175 5523 consensus_queue.cc:237] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:08.813614 5283 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:08.819465 5283 raft_consensus.cc:515] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:08.821035 5283 leader_election.cc:290] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:08.820858 5106 catalog_manager.cc:5696] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 2 to 3, leader changed from e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:08.821931 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:57:08.822057 5356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:08.822454 5209 raft_consensus.cc:3055] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:08.822585 5356 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:08.822854 5209 raft_consensus.cc:740] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:57:08.823655 5209 consensus_queue.cc:260] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:08.825023 5209 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:08.828801 5356 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:08.829844 5209 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:08.829946 5248 leader_election.cc:304] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:57:08.830772 5523 raft_consensus.cc:2804] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:08.831265 5523 raft_consensus.cc:697] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:08.832063 5523 consensus_queue.cc:237] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:08.841626 5106 catalog_manager.cc:5696] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 2 to 3, leader changed from e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:08.998919 5412 raft_consensus.cc:993] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:08.999337 5412 raft_consensus.cc:1081] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:09.000602 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37114
I20260620 19:57:09.001034 5283 raft_consensus.cc:493] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:09.001312 5283 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:09.005237 5283 raft_consensus.cc:515] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:57:09.006759 5283 leader_election.cc:290] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:09.007625 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:57:09.007763 5356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:09.008087 5209 raft_consensus.cc:3055] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:09.008234 5356 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:09.008389 5209 raft_consensus.cc:740] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:57:09.009011 5209 consensus_queue.cc:260] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:57:09.009805 5209 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:09.012554 5356 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:57:09.013510 5248 leader_election.cc:304] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, 9f431a428f5c4feb98f2c984e13311a7; no voters:
I20260620 19:57:09.013737 5209 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 2.
I20260620 19:57:09.014310 5523 raft_consensus.cc:2804] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:09.014786 5523 raft_consensus.cc:697] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:09.015538 5523 consensus_queue.cc:237] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:57:09.021405 5106 catalog_manager.cc:5696] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 1 to 2, leader changed from e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:09.124187 5519 raft_consensus.cc:993] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:09.124722 5535 raft_consensus.cc:1081] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:09.126073 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "77e89736f48a46978fd607ea294db738"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37104
I20260620 19:57:09.126502 5283 raft_consensus.cc:493] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:09.126821 5283 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:09.130790 5283 raft_consensus.cc:515] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:09.132252 5283 leader_election.cc:290] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:09.132947 5209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:57:09.133211 5356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:09.133409 5209 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:09.133723 5356 raft_consensus.cc:3055] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:09.133973 5356 raft_consensus.cc:740] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:57:09.134682 5356 consensus_queue.cc:260] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:09.135681 5356 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:09.137547 5209 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:09.138502 5249 leader_election.cc:304] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:57:09.139249 5523 raft_consensus.cc:2804] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:09.139645 5523 raft_consensus.cc:697] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:09.139860 5356 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:09.140501 5523 consensus_queue.cc:237] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:09.147279 5106 catalog_manager.cc:5696] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 2 to 3, leader changed from 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:09.215562 5519 raft_consensus.cc:993] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7: : Instructing follower 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:09.216019 5519 raft_consensus.cc:1081] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Signalling peer 8a5e3300aab24352be03d75663278916 to start an election
I20260620 19:57:09.217664 5283 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce"
dest_uuid: "8a5e3300aab24352be03d75663278916"
from {username='slave'} at 127.0.0.1:37104
I20260620 19:57:09.218031 5209 raft_consensus.cc:1275] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.218160 5283 raft_consensus.cc:493] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:09.218624 5283 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:09.219317 5523 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:09.222498 5208 raft_consensus.cc:1275] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.224325 5527 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:57:09.227248 5283 raft_consensus.cc:515] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:09.231709 5207 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:57:09.232527 5207 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:09.233788 5283 leader_election.cc:290] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:09.230258 5356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "8a5e3300aab24352be03d75663278916" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:09.235374 5356 raft_consensus.cc:3055] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:09.235749 5356 raft_consensus.cc:740] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:57:09.236789 5356 consensus_queue.cc:260] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:09.238221 5356 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:09.241000 5357 raft_consensus.cc:1275] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.242480 5527 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:09.244738 5207 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:09.246083 5249 leader_election.cc:304] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:57:09.247596 5527 raft_consensus.cc:2804] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:09.246054 5356 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5e3300aab24352be03d75663278916 in term 3.
I20260620 19:57:09.254741 5527 raft_consensus.cc:697] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Leader. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:09.255930 5527 consensus_queue.cc:237] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:09.257462 5356 raft_consensus.cc:1275] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.258713 5523 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:09.273607 5106 catalog_manager.cc:5696] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 reported cstate change: term changed from 2 to 3, leader changed from 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131) to 8a5e3300aab24352be03d75663278916 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "8a5e3300aab24352be03d75663278916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:09.399017 5356 raft_consensus.cc:1275] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:09.400460 5527 consensus_queue.cc:1048] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:09.410405 5208 raft_consensus.cc:1275] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:09.411712 5527 consensus_queue.cc:1048] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:09.529392 5356 raft_consensus.cc:1275] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.530648 5527 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:09.539402 5208 raft_consensus.cc:1275] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.541101 5543 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:09.701630 5208 raft_consensus.cc:1275] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.703150 5543 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:09.711169 5356 raft_consensus.cc:1275] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Refusing update from remote peer 8a5e3300aab24352be03d75663278916: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:09.712628 5523 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:11.793208 5074 auto_leader_rebalancer-test.cc:328] The leader distribution is
I20260620 19:57:11.794277 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:11.801388 5283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:54498
I20260620 19:57:11.801987 5283 raft_consensus.cc:606] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Received request to transfer leadership to TS e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:57:11.804733 5283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:54498
I20260620 19:57:11.805150 5283 raft_consensus.cc:606] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Received request to transfer leadership to TS 9f431a428f5c4feb98f2c984e13311a7
I20260620 19:57:11.807243 5283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "77e89736f48a46978fd607ea294db738"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:54498
I20260620 19:57:11.807583 5283 raft_consensus.cc:606] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Received request to transfer leadership to TS e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:57:11.809480 5283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:54498
I20260620 19:57:11.809861 5283 raft_consensus.cc:606] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Received request to transfer leadership to TS e2c0f888513f4dbca42e2df83cee8a02
I20260620 19:57:11.812175 5283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238"
dest_uuid: "8a5e3300aab24352be03d75663278916"
mode: GRACEFUL
new_leader_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:54498
I20260620 19:57:11.812567 5283 raft_consensus.cc:606] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Received request to transfer leadership to TS 9f431a428f5c4feb98f2c984e13311a7
I20260620 19:57:11.813515 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260620 19:57:11.813922 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:12.016003 5555 raft_consensus.cc:993] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: : Instructing follower e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:57:12.016449 5555 raft_consensus.cc:1081] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Signalling peer e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:57:12.017949 5208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:57822
I20260620 19:57:12.018448 5208 raft_consensus.cc:493] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:12.018859 5208 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.023924 5208 raft_consensus.cc:515] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.026032 5208 leader_election.cc:290] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 4 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:12.026790 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:57:12.027545 5283 raft_consensus.cc:3055] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:57:12.027460 5357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a086e8da4c43a0b3ef18242b74bf2f" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:12.027871 5283 raft_consensus.cc:740] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:12.028026 5357 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.028743 5283 consensus_queue.cc:260] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.029964 5283 raft_consensus.cc:3060] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.034492 5357 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 4.
I20260620 19:57:12.035909 5173 leader_election.cc:304] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:57:12.036335 5283 raft_consensus.cc:2468] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 4.
I20260620 19:57:12.036602 5527 raft_consensus.cc:993] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916: : Instructing follower e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:57:12.037217 5527 raft_consensus.cc:1081] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Signalling peer e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:57:12.037598 5556 raft_consensus.cc:2804] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:57:12.038811 5208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "77e89736f48a46978fd607ea294db738"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:57822
I20260620 19:57:12.038956 5556 raft_consensus.cc:697] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 4 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:57:12.039393 5208 raft_consensus.cc:493] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:12.039911 5208 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.040242 5556 consensus_queue.cc:237] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.047174 5559 raft_consensus.cc:993] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916: : Instructing follower 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:57:12.046382 5208 raft_consensus.cc:515] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.047683 5559 raft_consensus.cc:1081] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Signalling peer 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:57:12.049335 5357 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:35788
I20260620 19:57:12.049572 5283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:57:12.049122 5106 catalog_manager.cc:5696] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 3 to 4, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:12.050050 5283 raft_consensus.cc:3055] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:57:12.049849 5357 raft_consensus.cc:493] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:12.050335 5208 leader_election.cc:290] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 4 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:12.050474 5357 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:12.051249 5356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77e89736f48a46978fd607ea294db738" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:12.051770 5356 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.050348 5283 raft_consensus.cc:740] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:12.052734 5283 consensus_queue.cc:260] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.053790 5283 raft_consensus.cc:3060] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.056742 5357 raft_consensus.cc:515] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:57:12.059526 5357 leader_election.cc:290] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 3 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:57:12.059819 5283 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 4.
I20260620 19:57:12.061133 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:57:12.061239 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb69c1c8316344a2bc9557bb2ce94238" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:57:12.061671 5208 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:12.061954 5284 raft_consensus.cc:3055] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:12.062050 5356 raft_consensus.cc:2468] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 4.
I20260620 19:57:12.062400 5284 raft_consensus.cc:740] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:12.063398 5284 consensus_queue.cc:260] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:57:12.064185 5284 raft_consensus.cc:3060] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:12.067662 5172 leader_election.cc:304] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5e3300aab24352be03d75663278916, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:57:12.068063 5208 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 3.
I20260620 19:57:12.068696 5556 raft_consensus.cc:2804] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:57:12.069218 5556 raft_consensus.cc:697] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:57:12.069485 5323 leader_election.cc:304] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:57:12.070021 5556 consensus_queue.cc:237] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.070806 5535 raft_consensus.cc:2804] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:12.071373 5284 raft_consensus.cc:2468] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 3.
I20260620 19:57:12.073040 5535 raft_consensus.cc:697] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 LEADER]: Becoming Leader. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:57:12.074194 5535 consensus_queue.cc:237] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } }
I20260620 19:57:12.082700 5106 catalog_manager.cc:5696] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 reported cstate change: term changed from 2 to 3, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "9f431a428f5c4feb98f2c984e13311a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:12.084336 5102 catalog_manager.cc:5696] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 3 to 4, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:12.133000 5527 raft_consensus.cc:993] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916: : Instructing follower 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:57:12.133438 5555 raft_consensus.cc:1081] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Signalling peer 9f431a428f5c4feb98f2c984e13311a7 to start an election
I20260620 19:57:12.134785 5356 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53"
dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
from {username='slave'} at 127.0.0.1:35788
I20260620 19:57:12.135214 5356 raft_consensus.cc:493] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:12.135468 5356 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.139480 5356 raft_consensus.cc:515] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.141130 5356 leader_election.cc:290] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 4 election: Requested vote from peers e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041), 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111)
I20260620 19:57:12.141852 5208 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
I20260620 19:57:12.142109 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d070e48b2424cc480e243cce4e7fc53" candidate_uuid: "9f431a428f5c4feb98f2c984e13311a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:57:12.142405 5208 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.142645 5284 raft_consensus.cc:3055] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:57:12.142954 5284 raft_consensus.cc:740] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:12.143666 5284 consensus_queue.cc:260] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.144503 5284 raft_consensus.cc:3060] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.146854 5208 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 4.
I20260620 19:57:12.147737 5323 leader_election.cc:304] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:57:12.148409 5535 raft_consensus.cc:2804] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:57:12.148561 5284 raft_consensus.cc:2468] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f431a428f5c4feb98f2c984e13311a7 in term 4.
I20260620 19:57:12.148797 5535 raft_consensus.cc:697] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 LEADER]: Becoming Leader. State: Replica: 9f431a428f5c4feb98f2c984e13311a7, State: Running, Role: LEADER
I20260620 19:57:12.149937 5535 consensus_queue.cc:237] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.156140 5102 catalog_manager.cc:5696] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 reported cstate change: term changed from 3 to 4, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131). New cstate: current_term: 4 leader_uuid: "9f431a428f5c4feb98f2c984e13311a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:12.167351 5527 raft_consensus.cc:993] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: : Instructing follower e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:57:12.167747 5555 raft_consensus.cc:1081] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Signalling peer e2c0f888513f4dbca42e2df83cee8a02 to start an election
I20260620 19:57:12.169226 5208 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce"
dest_uuid: "e2c0f888513f4dbca42e2df83cee8a02"
from {username='slave'} at 127.0.0.1:57822
I20260620 19:57:12.169656 5208 raft_consensus.cc:493] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:12.169909 5208 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.173643 5208 raft_consensus.cc:515] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.174986 5208 leader_election.cc:290] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 4 election: Requested vote from peers 8a5e3300aab24352be03d75663278916 (127.4.244.130:38111), 9f431a428f5c4feb98f2c984e13311a7 (127.4.244.131:35885)
I20260620 19:57:12.175712 5284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8a5e3300aab24352be03d75663278916"
I20260620 19:57:12.176007 5356 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c3ac57b2aec4191ae5f5f818c07c1ce" candidate_uuid: "e2c0f888513f4dbca42e2df83cee8a02" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f431a428f5c4feb98f2c984e13311a7"
I20260620 19:57:12.176219 5284 raft_consensus.cc:3055] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:57:12.176499 5356 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.176512 5284 raft_consensus.cc:740] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8a5e3300aab24352be03d75663278916, State: Running, Role: LEADER
I20260620 19:57:12.177390 5284 consensus_queue.cc:260] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.178494 5284 raft_consensus.cc:3060] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:12.182572 5356 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 4.
I20260620 19:57:12.183681 5173 leader_election.cc:304] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f431a428f5c4feb98f2c984e13311a7, e2c0f888513f4dbca42e2df83cee8a02; no voters:
I20260620 19:57:12.184137 5284 raft_consensus.cc:2468] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c0f888513f4dbca42e2df83cee8a02 in term 4.
I20260620 19:57:12.184350 5556 raft_consensus.cc:2804] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:57:12.184800 5556 raft_consensus.cc:697] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 4 LEADER]: Becoming Leader. State: Replica: e2c0f888513f4dbca42e2df83cee8a02, State: Running, Role: LEADER
I20260620 19:57:12.185616 5556 consensus_queue.cc:237] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } }
I20260620 19:57:12.192111 5102 catalog_manager.cc:5696] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 reported cstate change: term changed from 3 to 4, leader changed from 8a5e3300aab24352be03d75663278916 (127.4.244.130) to e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "e2c0f888513f4dbca42e2df83cee8a02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:12.511636 5284 raft_consensus.cc:1275] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.512796 5556 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.522065 5356 raft_consensus.cc:1275] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.524127 5556 consensus_queue.cc:1048] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.544323 5284 raft_consensus.cc:1275] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:12.545619 5535 consensus_queue.cc:1048] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:12.554109 5208 raft_consensus.cc:1275] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:12.555804 5535 consensus_queue.cc:1048] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:12.577013 5284 raft_consensus.cc:1275] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.578039 5556 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.586369 5356 raft_consensus.cc:1275] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.587962 5556 consensus_queue.cc:1048] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.638036 5208 raft_consensus.cc:1275] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.639132 5575 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0f888513f4dbca42e2df83cee8a02" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.647889 5284 raft_consensus.cc:1275] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Refusing update from remote peer 9f431a428f5c4feb98f2c984e13311a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.649513 5575 consensus_queue.cc:1048] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.673058 5284 raft_consensus.cc:1275] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.674113 5557 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a5e3300aab24352be03d75663278916" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.681854 5357 raft_consensus.cc:1275] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Refusing update from remote peer e2c0f888513f4dbca42e2df83cee8a02: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:12.683292 5557 consensus_queue.cc:1048] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f431a428f5c4feb98f2c984e13311a7" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:12.815126 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:12.819741 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:12.820113 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:13.821027 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:13.825603 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:13.825997 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:14.827145 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:14.831846 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:14.832242 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:15.833194 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:15.837872 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:15.838240 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:16.839154 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:16.843864 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:16.844250 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:17.845170 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:17.849746 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:17.850127 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:18.851135 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:18.856030 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:18.856396 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:19.857316 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:19.862576 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:19.862972 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:20.863955 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:20.868840 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:20.869233 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:21.896155 5074 auto_leader_rebalancer-test.cc:340] The leader distribution is
I20260620 19:57:21.896770 5074 tablet_server.cc:179] TabletServer@127.4.244.129:0 shutting down...
W20260620 19:57:21.924588 5323 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260620 19:57:21.929052 5323 consensus_peers.cc:597] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 -> Peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Couldn't send request to peer e2c0f888513f4dbca42e2df83cee8a02. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:21.929776 5074 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260620 19:57:21.930689 5074 tablet_replica.cc:333] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.931468 5074 raft_consensus.cc:2243] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 4 LEADER]: Raft consensus shutting down.
I20260620 19:57:21.932425 5074 raft_consensus.cc:2272] T 2c3ac57b2aec4191ae5f5f818c07c1ce P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.934967 5074 tablet_replica.cc:333] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.935453 5074 raft_consensus.cc:2243] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 LEADER]: Raft consensus shutting down.
I20260620 19:57:21.936158 5074 raft_consensus.cc:2272] T 77e89736f48a46978fd607ea294db738 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.937769 5074 tablet_replica.cc:333] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.938247 5074 raft_consensus.cc:2243] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:21.938792 5074 raft_consensus.cc:2272] T f1ba73a8009d466f9db25d83eec1af67 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.940536 5074 tablet_replica.cc:333] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.941035 5074 raft_consensus.cc:2243] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 4 LEADER]: Raft consensus shutting down.
I20260620 19:57:21.941761 5074 raft_consensus.cc:2272] T c9a086e8da4c43a0b3ef18242b74bf2f P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.943493 5074 tablet_replica.cc:333] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.943961 5074 raft_consensus.cc:2243] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:21.944368 5074 raft_consensus.cc:2272] T 7066f7b456e4438e93cd393e270a554b P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.946173 5074 tablet_replica.cc:333] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.946851 5074 raft_consensus.cc:2243] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:21.947471 5074 raft_consensus.cc:2272] T fb69c1c8316344a2bc9557bb2ce94238 P e2c0f888513f4dbca42e2df83cee8a02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.949842 5074 tablet_replica.cc:333] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
W20260620 19:57:21.950158 5249 consensus_peers.cc:597] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 -> Peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Couldn't send request to peer e2c0f888513f4dbca42e2df83cee8a02. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:21.950541 5074 raft_consensus.cc:2243] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:21.951020 5074 raft_consensus.cc:2272] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.952600 5074 tablet_replica.cc:333] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.953006 5074 raft_consensus.cc:2243] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:21.953388 5074 raft_consensus.cc:2272] T 4d070e48b2424cc480e243cce4e7fc53 P e2c0f888513f4dbca42e2df83cee8a02 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:21.955044 5074 tablet_replica.cc:333] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02: stopping tablet replica
I20260620 19:57:21.955435 5074 raft_consensus.cc:2243] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:21.955780 5074 raft_consensus.cc:2272] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P e2c0f888513f4dbca42e2df83cee8a02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.041394 5074 tablet_server.cc:196] TabletServer@127.4.244.129:0 shutdown complete.
W20260620 19:57:22.042279 5323 consensus_peers.cc:597] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 -> Peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Couldn't send request to peer e2c0f888513f4dbca42e2df83cee8a02. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 19:57:22.049450 5249 consensus_peers.cc:597] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 -> Peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Couldn't send request to peer e2c0f888513f4dbca42e2df83cee8a02. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:22.067322 5074 tablet_server.cc:179] TabletServer@127.4.244.130:0 shutting down...
I20260620 19:57:22.091315 5074 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260620 19:57:22.091980 5074 tablet_replica.cc:333] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.092538 5074 raft_consensus.cc:2243] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.093009 5074 raft_consensus.cc:2272] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.094810 5074 tablet_replica.cc:333] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.095310 5074 raft_consensus.cc:2243] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Raft consensus shutting down.
I20260620 19:57:22.096261 5074 raft_consensus.cc:2272] T 7066f7b456e4438e93cd393e270a554b P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.097931 5074 tablet_replica.cc:333] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.098419 5074 raft_consensus.cc:2243] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.098847 5074 raft_consensus.cc:2272] T 77e89736f48a46978fd607ea294db738 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.100322 5074 tablet_replica.cc:333] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.100778 5074 raft_consensus.cc:2243] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Raft consensus shutting down.
I20260620 19:57:22.101475 5074 raft_consensus.cc:2272] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.103220 5074 tablet_replica.cc:333] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.103708 5074 raft_consensus.cc:2243] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.104094 5074 raft_consensus.cc:2272] T fb69c1c8316344a2bc9557bb2ce94238 P 8a5e3300aab24352be03d75663278916 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.105567 5074 tablet_replica.cc:333] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.106135 5074 raft_consensus.cc:2243] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.106527 5074 raft_consensus.cc:2272] T f1ba73a8009d466f9db25d83eec1af67 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.107968 5074 tablet_replica.cc:333] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.108417 5074 raft_consensus.cc:2243] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.108804 5074 raft_consensus.cc:2272] T c9a086e8da4c43a0b3ef18242b74bf2f P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.110267 5074 tablet_replica.cc:333] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.110749 5074 raft_consensus.cc:2243] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 2 LEADER]: Raft consensus shutting down.
I20260620 19:57:22.111460 5074 raft_consensus.cc:2272] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 8a5e3300aab24352be03d75663278916 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.112875 5074 tablet_replica.cc:333] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916: stopping tablet replica
I20260620 19:57:22.113291 5074 raft_consensus.cc:2243] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.113711 5074 raft_consensus.cc:2272] T 4d070e48b2424cc480e243cce4e7fc53 P 8a5e3300aab24352be03d75663278916 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.140628 5074 tablet_server.cc:196] TabletServer@127.4.244.130:0 shutdown complete.
I20260620 19:57:22.158250 5074 tablet_server.cc:179] TabletServer@127.4.244.131:0 shutting down...
I20260620 19:57:22.187354 5074 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260620 19:57:22.188241 5074 tablet_replica.cc:333] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.189169 5074 raft_consensus.cc:2243] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.189826 5074 raft_consensus.cc:2272] T 7066f7b456e4438e93cd393e270a554b P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.191877 5074 tablet_replica.cc:333] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
W20260620 19:57:22.192148 5323 consensus_peers.cc:597] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 -> Peer e2c0f888513f4dbca42e2df83cee8a02 (127.4.244.129:34041): Couldn't send request to peer e2c0f888513f4dbca42e2df83cee8a02. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:22.192474 5074 raft_consensus.cc:2243] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.192941 5074 raft_consensus.cc:2272] T 2c3ac57b2aec4191ae5f5f818c07c1ce P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.194510 5074 tablet_replica.cc:333] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.195062 5074 raft_consensus.cc:2243] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 LEADER]: Raft consensus shutting down.
I20260620 19:57:22.195767 5074 raft_consensus.cc:2272] T 4d070e48b2424cc480e243cce4e7fc53 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.197278 5074 tablet_replica.cc:333] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.197773 5074 raft_consensus.cc:2243] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.198182 5074 raft_consensus.cc:2272] T 77e89736f48a46978fd607ea294db738 P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.200196 5074 tablet_replica.cc:333] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.200661 5074 raft_consensus.cc:2243] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.201050 5074 raft_consensus.cc:2272] T 0fa02cada6aa4a179ef372d4ce3a0fc9 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.202530 5074 tablet_replica.cc:333] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.203033 5074 raft_consensus.cc:2243] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.203418 5074 raft_consensus.cc:2272] T 6a5d1fb6c9b0439d99aa9c8447b33ed3 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.204906 5074 tablet_replica.cc:333] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.205377 5074 raft_consensus.cc:2243] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 LEADER]: Raft consensus shutting down.
I20260620 19:57:22.206137 5074 raft_consensus.cc:2272] T fb69c1c8316344a2bc9557bb2ce94238 P 9f431a428f5c4feb98f2c984e13311a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.207679 5074 tablet_replica.cc:333] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.208134 5074 raft_consensus.cc:2243] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 LEADER]: Raft consensus shutting down.
I20260620 19:57:22.208823 5074 raft_consensus.cc:2272] T f1ba73a8009d466f9db25d83eec1af67 P 9f431a428f5c4feb98f2c984e13311a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.210316 5074 tablet_replica.cc:333] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7: stopping tablet replica
I20260620 19:57:22.210814 5074 raft_consensus.cc:2243] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:22.211239 5074 raft_consensus.cc:2272] T c9a086e8da4c43a0b3ef18242b74bf2f P 9f431a428f5c4feb98f2c984e13311a7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.237299 5074 tablet_server.cc:196] TabletServer@127.4.244.131:0 shutdown complete.
I20260620 19:57:22.256295 5074 master.cc:562] Master@127.4.244.190:36613 shutting down...
I20260620 19:57:22.387521 5074 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:57:22.388092 5074 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:22.388367 5074 tablet_replica.cc:333] T 00000000000000000000000000000000 P 288e961cf54049eba9a0909a850e73dc: stopping tablet replica
I20260620 19:57:22.406001 5074 master.cc:584] Master@127.4.244.190:36613 shutdown complete.
8a5e3300aab24352be03d75663278916 4
9f431a428f5c4feb98f2c984e13311a7 1
e2c0f888513f4dbca42e2df83cee8a02 4
8a5e3300aab24352be03d75663278916 3
9f431a428f5c4feb98f2c984e13311a7 3
e2c0f888513f4dbca42e2df83cee8a02 3
8a5e3300aab24352be03d75663278916 8
9f431a428f5c4feb98f2c984e13311a7 1
e2c0f888513f4dbca42e2df83cee8a02 0
8a5e3300aab24352be03d75663278916 3
9f431a428f5c4feb98f2c984e13311a7 3
e2c0f888513f4dbca42e2df83cee8a02 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31184 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260620 19:57:22.448671 5074 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.244.190:33981
I20260620 19:57:22.449803 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:22.455518 5598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:22.455644 5599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:22.457360 5601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:22.457751 5074 server_base.cc:1061] running on GCE node
I20260620 19:57:22.458940 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:22.459160 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:22.459323 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985442459303 us; error 0 us; skew 500 ppm
I20260620 19:57:22.459841 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:22.462225 5074 webserver.cc:533] Webserver started at http://127.4.244.190:33391/ using document root <none> and password file <none>
I20260620 19:57:22.462764 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:22.462966 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:22.463238 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:22.464390 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/master-0-root/instance:
uuid: "16b6b364cb1249c2a1b9068751bcde4e"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.468716 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:57:22.471932 5606 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.472759 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:57:22.472983 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/master-0-root
uuid: "16b6b364cb1249c2a1b9068751bcde4e"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.473207 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:22.493391 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:22.494510 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:22.530108 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.190:33981
I20260620 19:57:22.530284 5657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.190:33981 every 8 connection(s)
I20260620 19:57:22.534106 5658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:22.547359 5658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e: Bootstrap starting.
I20260620 19:57:22.551901 5658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:22.555819 5658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e: No bootstrap required, opened a new log
I20260620 19:57:22.557870 5658 raft_consensus.cc:359] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6b364cb1249c2a1b9068751bcde4e" member_type: VOTER }
I20260620 19:57:22.558262 5658 raft_consensus.cc:385] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:22.558514 5658 raft_consensus.cc:740] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b6b364cb1249c2a1b9068751bcde4e, State: Initialized, Role: FOLLOWER
I20260620 19:57:22.559171 5658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6b364cb1249c2a1b9068751bcde4e" member_type: VOTER }
I20260620 19:57:22.559655 5658 raft_consensus.cc:399] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:57:22.559899 5658 raft_consensus.cc:493] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:57:22.560156 5658 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:22.564044 5658 raft_consensus.cc:515] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6b364cb1249c2a1b9068751bcde4e" member_type: VOTER }
I20260620 19:57:22.564534 5658 leader_election.cc:304] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b6b364cb1249c2a1b9068751bcde4e; no voters:
I20260620 19:57:22.565608 5658 leader_election.cc:290] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:57:22.565999 5661 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:22.567252 5661 raft_consensus.cc:697] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 1 LEADER]: Becoming Leader. State: Replica: 16b6b364cb1249c2a1b9068751bcde4e, State: Running, Role: LEADER
I20260620 19:57:22.567916 5661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6b364cb1249c2a1b9068751bcde4e" member_type: VOTER }
I20260620 19:57:22.568409 5658 sys_catalog.cc:565] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:57:22.572572 5663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b6b364cb1249c2a1b9068751bcde4e. Latest consensus state: current_term: 1 leader_uuid: "16b6b364cb1249c2a1b9068751bcde4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6b364cb1249c2a1b9068751bcde4e" member_type: VOTER } }
I20260620 19:57:22.573185 5663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [sys.catalog]: This master's current role is: LEADER
I20260620 19:57:22.574652 5662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b6b364cb1249c2a1b9068751bcde4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b6b364cb1249c2a1b9068751bcde4e" member_type: VOTER } }
I20260620 19:57:22.575187 5668 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 19:57:22.575493 5662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [sys.catalog]: This master's current role is: LEADER
I20260620 19:57:22.580507 5668 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 19:57:22.585501 5074 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260620 19:57:22.590278 5668 catalog_manager.cc:1382] Generated new cluster ID: 382b0246d914436195bcfa4abcd08f40
I20260620 19:57:22.590548 5668 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 19:57:22.602026 5668 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 19:57:22.603185 5668 catalog_manager.cc:1539] Loading token signing keys...
I20260620 19:57:22.617550 5668 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e: Generated new TSK 0
I20260620 19:57:22.618249 5668 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 19:57:22.651678 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:22.657488 5679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:22.658735 5680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:22.660979 5074 server_base.cc:1061] running on GCE node
W20260620 19:57:22.661348 5682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:22.662190 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:22.662447 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:22.662662 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985442662641 us; error 0 us; skew 500 ppm
I20260620 19:57:22.663172 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:22.665264 5074 webserver.cc:533] Webserver started at http://127.4.244.129:35011/ using document root <none> and password file <none>
I20260620 19:57:22.665727 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:22.665922 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:22.666188 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:22.667276 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-0-root/instance:
uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.671622 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:57:22.674657 5687 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.675455 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:57:22.675755 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-0-root
uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.675992 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:22.695639 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:22.696802 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:22.698208 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:57:22.700340 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:57:22.700548 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.700793 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:57:22.700977 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.740783 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.129:34991
I20260620 19:57:22.740900 5749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.129:34991 every 8 connection(s)
I20260620 19:57:22.745481 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:22.752976 5754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:22.753302 5755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:22.757577 5757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:22.758479 5074 server_base.cc:1061] running on GCE node
I20260620 19:57:22.758821 5750 heartbeater.cc:344] Connected to a master server at 127.4.244.190:33981
I20260620 19:57:22.759232 5750 heartbeater.cc:461] Registering TS with master...
I20260620 19:57:22.759469 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:22.759702 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:22.759876 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985442759851 us; error 0 us; skew 500 ppm
I20260620 19:57:22.759990 5750 heartbeater.cc:507] Master 127.4.244.190:33981 requested a full tablet report, sending...
I20260620 19:57:22.760427 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:22.762233 5623 ts_manager.cc:194] Registered new tserver with Master: dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991)
I20260620 19:57:22.763471 5074 webserver.cc:533] Webserver started at http://127.4.244.130:38613/ using document root <none> and password file <none>
I20260620 19:57:22.764138 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:22.764382 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:22.764690 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:22.764880 5623 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50520
I20260620 19:57:22.766254 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-1-root/instance:
uuid: "994e5d10524b4801a8457e3a58f2527d"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.770781 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:57:22.773976 5762 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.774854 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:57:22.775152 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-1-root
uuid: "994e5d10524b4801a8457e3a58f2527d"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.775444 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:22.784934 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:22.786118 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:22.787655 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:57:22.789844 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:57:22.790083 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.790330 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:57:22.790508 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.829530 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.130:33843
I20260620 19:57:22.829623 5824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.130:33843 every 8 connection(s)
I20260620 19:57:22.834280 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:22.840950 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:22.843441 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:22.846727 5825 heartbeater.cc:344] Connected to a master server at 127.4.244.190:33981
I20260620 19:57:22.847162 5825 heartbeater.cc:461] Registering TS with master...
W20260620 19:57:22.847452 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:22.848043 5074 server_base.cc:1061] running on GCE node
I20260620 19:57:22.848068 5825 heartbeater.cc:507] Master 127.4.244.190:33981 requested a full tablet report, sending...
I20260620 19:57:22.849156 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:22.849392 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:22.849598 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985442849578 us; error 0 us; skew 500 ppm
I20260620 19:57:22.850342 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:22.850562 5623 ts_manager.cc:194] Registered new tserver with Master: 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:22.852455 5623 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50530
I20260620 19:57:22.853019 5074 webserver.cc:533] Webserver started at http://127.4.244.131:46375/ using document root <none> and password file <none>
I20260620 19:57:22.853605 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:22.853859 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:22.854182 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:22.855559 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-2-root/instance:
uuid: "cb9a18ec75324292b1c71c978353bfed"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.859733 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:57:22.862775 5836 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.863508 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:57:22.863806 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-2-root
uuid: "cb9a18ec75324292b1c71c978353bfed"
format_stamp: "Formatted at 2026-06-20 19:57:22 on dist-test-slave-zxgc"
I20260620 19:57:22.864111 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1781985411095902-5074-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:22.873809 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:22.875136 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:22.876533 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:57:22.878907 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:57:22.879206 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.879473 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:57:22.879660 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:22.918031 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.131:34813
I20260620 19:57:22.918118 5898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.131:34813 every 8 connection(s)
I20260620 19:57:22.931952 5899 heartbeater.cc:344] Connected to a master server at 127.4.244.190:33981
I20260620 19:57:22.932298 5899 heartbeater.cc:461] Registering TS with master...
I20260620 19:57:22.932921 5899 heartbeater.cc:507] Master 127.4.244.190:33981 requested a full tablet report, sending...
I20260620 19:57:22.934788 5623 ts_manager.cc:194] Registered new tserver with Master: cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:22.935436 5074 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014361116s
I20260620 19:57:22.935995 5623 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50536
I20260620 19:57:22.959753 5623 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50550:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:57:22.962499 5623 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:57:23.037285 5715 tablet_service.cc:1511] Processing CreateTablet for tablet 8e87edcc1a1a44c987e7077250420b69 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260620 19:57:23.037609 5714 tablet_service.cc:1511] Processing CreateTablet for tablet f90568e63c88430ea023f53fb89ac8d4 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260620 19:57:23.038841 5715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e87edcc1a1a44c987e7077250420b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.039412 5713 tablet_service.cc:1511] Processing CreateTablet for tablet 5271c718decc4e10895728893bf42fcf (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260620 19:57:23.040938 5713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5271c718decc4e10895728893bf42fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.043232 5714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f90568e63c88430ea023f53fb89ac8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.050110 5864 tablet_service.cc:1511] Processing CreateTablet for tablet 8e87edcc1a1a44c987e7077250420b69 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260620 19:57:23.050904 5862 tablet_service.cc:1511] Processing CreateTablet for tablet 5271c718decc4e10895728893bf42fcf (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260620 19:57:23.051584 5864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e87edcc1a1a44c987e7077250420b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.052256 5862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5271c718decc4e10895728893bf42fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.055861 5863 tablet_service.cc:1511] Processing CreateTablet for tablet f90568e63c88430ea023f53fb89ac8d4 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260620 19:57:23.057251 5863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f90568e63c88430ea023f53fb89ac8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.061096 5914 tablet_bootstrap.cc:492] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.067520 5914 tablet_bootstrap.cc:654] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.070708 5861 tablet_service.cc:1511] Processing CreateTablet for tablet 16dfb3ea9ba94e66a939bfea5adb58ff (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260620 19:57:23.071404 5712 tablet_service.cc:1511] Processing CreateTablet for tablet 965976254d4a461ba6ad6af4aaa5c441 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260620 19:57:23.072165 5861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16dfb3ea9ba94e66a939bfea5adb58ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.073123 5712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965976254d4a461ba6ad6af4aaa5c441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.078938 5914 tablet_bootstrap.cc:492] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.079579 5914 ts_tablet_manager.cc:1403] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20260620 19:57:23.082448 5914 raft_consensus.cc:359] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.083415 5914 raft_consensus.cc:385] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.083751 5914 raft_consensus.cc:740] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.084688 5914 consensus_queue.cc:260] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.094202 5914 ts_tablet_manager.cc:1434] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.014s user 0.005s sys 0.006s
I20260620 19:57:23.066723 5715 tablet_service.cc:1511] Processing CreateTablet for tablet 16dfb3ea9ba94e66a939bfea5adb58ff (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260620 19:57:23.095014 5750 heartbeater.cc:499] Master 127.4.244.190:33981 was elected leader, sending a full tablet report...
I20260620 19:57:23.095696 5715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16dfb3ea9ba94e66a939bfea5adb58ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.097113 5790 tablet_service.cc:1511] Processing CreateTablet for tablet 8e87edcc1a1a44c987e7077250420b69 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260620 19:57:23.098316 5790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e87edcc1a1a44c987e7077250420b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.099735 5789 tablet_service.cc:1511] Processing CreateTablet for tablet f90568e63c88430ea023f53fb89ac8d4 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260620 19:57:23.101891 5788 tablet_service.cc:1511] Processing CreateTablet for tablet 5271c718decc4e10895728893bf42fcf (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260620 19:57:23.104072 5787 tablet_service.cc:1511] Processing CreateTablet for tablet 16dfb3ea9ba94e66a939bfea5adb58ff (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260620 19:57:23.108716 5789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f90568e63c88430ea023f53fb89ac8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.115010 5788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5271c718decc4e10895728893bf42fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.118999 5787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16dfb3ea9ba94e66a939bfea5adb58ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.129479 5918 tablet_bootstrap.cc:492] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.140311 5711 tablet_service.cc:1511] Processing CreateTablet for tablet b7123643714146939d0d82713a70121e (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260620 19:57:23.134713 5790 tablet_service.cc:1511] Processing CreateTablet for tablet 965976254d4a461ba6ad6af4aaa5c441 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260620 19:57:23.141700 5711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7123643714146939d0d82713a70121e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.141726 5790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965976254d4a461ba6ad6af4aaa5c441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.142875 5914 tablet_bootstrap.cc:492] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.149238 5861 tablet_service.cc:1511] Processing CreateTablet for tablet 965976254d4a461ba6ad6af4aaa5c441 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260620 19:57:23.151064 5914 tablet_bootstrap.cc:654] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.150547 5861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965976254d4a461ba6ad6af4aaa5c441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.154443 5786 tablet_service.cc:1511] Processing CreateTablet for tablet b7123643714146939d0d82713a70121e (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260620 19:57:23.157986 5786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7123643714146939d0d82713a70121e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.164696 5863 tablet_service.cc:1511] Processing CreateTablet for tablet b7123643714146939d0d82713a70121e (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260620 19:57:23.165975 5863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7123643714146939d0d82713a70121e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.168484 5918 tablet_bootstrap.cc:654] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.179039 5916 tablet_bootstrap.cc:492] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.183311 5914 tablet_bootstrap.cc:492] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.182926 5713 tablet_service.cc:1511] Processing CreateTablet for tablet f1814c9aa9f640b7bc73abee51e147c3 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260620 19:57:23.183799 5914 ts_tablet_manager.cc:1403] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.007s
I20260620 19:57:23.186866 5789 tablet_service.cc:1511] Processing CreateTablet for tablet f1814c9aa9f640b7bc73abee51e147c3 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260620 19:57:23.187584 5914 raft_consensus.cc:359] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.188103 5789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1814c9aa9f640b7bc73abee51e147c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.188310 5914 raft_consensus.cc:385] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.188623 5914 raft_consensus.cc:740] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.189342 5914 consensus_queue.cc:260] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.190284 5714 tablet_service.cc:1511] Processing CreateTablet for tablet e06c815cc3e64a9fb26c529634a70ccb (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260620 19:57:23.192263 5862 tablet_service.cc:1511] Processing CreateTablet for tablet f1814c9aa9f640b7bc73abee51e147c3 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260620 19:57:23.207024 5785 tablet_service.cc:1511] Processing CreateTablet for tablet f9112ebb71314faba51ce60093f7694f (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260620 19:57:23.208156 5862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1814c9aa9f640b7bc73abee51e147c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.208588 5785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9112ebb71314faba51ce60093f7694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.208777 5918 tablet_bootstrap.cc:492] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.209175 5918 ts_tablet_manager.cc:1403] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.014s
I20260620 19:57:23.211782 5860 tablet_service.cc:1511] Processing CreateTablet for tablet f9112ebb71314faba51ce60093f7694f (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260620 19:57:23.212049 5918 raft_consensus.cc:359] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.212772 5918 raft_consensus.cc:385] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.213038 5860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9112ebb71314faba51ce60093f7694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.213075 5918 raft_consensus.cc:740] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.213941 5918 consensus_queue.cc:260] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.194111 5916 tablet_bootstrap.cc:654] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.184688 5713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1814c9aa9f640b7bc73abee51e147c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.197063 5790 tablet_service.cc:1511] Processing CreateTablet for tablet e06c815cc3e64a9fb26c529634a70ccb (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260620 19:57:23.217146 5790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06c815cc3e64a9fb26c529634a70ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.199384 5864 tablet_service.cc:1511] Processing CreateTablet for tablet e06c815cc3e64a9fb26c529634a70ccb (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260620 19:57:23.202975 5712 tablet_service.cc:1511] Processing CreateTablet for tablet f9112ebb71314faba51ce60093f7694f (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260620 19:57:23.223553 5864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06c815cc3e64a9fb26c529634a70ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.227077 5712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9112ebb71314faba51ce60093f7694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.239184 5714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06c815cc3e64a9fb26c529634a70ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.243970 5914 ts_tablet_manager.cc:1434] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20260620 19:57:23.247073 5914 tablet_bootstrap.cc:492] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.252341 5711 tablet_service.cc:1511] Processing CreateTablet for tablet fa96289638f14c039a9f769c43a5d912 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260620 19:57:23.253600 5711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa96289638f14c039a9f769c43a5d912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.266335 5788 tablet_service.cc:1511] Processing CreateTablet for tablet fa96289638f14c039a9f769c43a5d912 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260620 19:57:23.267591 5788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa96289638f14c039a9f769c43a5d912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.283620 5864 tablet_service.cc:1511] Processing CreateTablet for tablet fa96289638f14c039a9f769c43a5d912 (DEFAULT_TABLE table=test-workload [id=3fa65455294047e2ab368550148dac29]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260620 19:57:23.284941 5864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa96289638f14c039a9f769c43a5d912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:23.287709 5825 heartbeater.cc:499] Master 127.4.244.190:33981 was elected leader, sending a full tablet report...
I20260620 19:57:23.288296 5918 ts_tablet_manager.cc:1434] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.079s user 0.027s sys 0.047s
I20260620 19:57:23.289232 5918 tablet_bootstrap.cc:492] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.292598 5916 tablet_bootstrap.cc:492] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.293080 5916 ts_tablet_manager.cc:1403] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.114s user 0.015s sys 0.000s
I20260620 19:57:23.295425 5918 tablet_bootstrap.cc:654] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.295894 5916 raft_consensus.cc:359] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.296641 5916 raft_consensus.cc:385] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.296938 5916 raft_consensus.cc:740] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.297811 5916 consensus_queue.cc:260] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.301012 5899 heartbeater.cc:499] Master 127.4.244.190:33981 was elected leader, sending a full tablet report...
I20260620 19:57:23.303481 5914 tablet_bootstrap.cc:654] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.307278 5918 tablet_bootstrap.cc:492] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.307732 5918 ts_tablet_manager.cc:1403] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20260620 19:57:23.310508 5918 raft_consensus.cc:359] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.311236 5918 raft_consensus.cc:385] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.311515 5918 raft_consensus.cc:740] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.312212 5918 consensus_queue.cc:260] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.314218 5918 ts_tablet_manager.cc:1434] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:57:23.315156 5918 tablet_bootstrap.cc:492] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.317063 5916 ts_tablet_manager.cc:1434] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20260620 19:57:23.318039 5916 tablet_bootstrap.cc:492] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.321811 5914 tablet_bootstrap.cc:492] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.322003 5918 tablet_bootstrap.cc:654] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.322515 5914 ts_tablet_manager.cc:1403] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.076s user 0.020s sys 0.005s
I20260620 19:57:23.325445 5916 tablet_bootstrap.cc:654] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.325671 5914 raft_consensus.cc:359] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.326545 5914 raft_consensus.cc:385] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.326923 5914 raft_consensus.cc:740] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.327716 5914 consensus_queue.cc:260] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.330312 5914 ts_tablet_manager.cc:1434] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:23.331403 5914 tablet_bootstrap.cc:492] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.338058 5914 tablet_bootstrap.cc:654] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.346786 5921 raft_consensus.cc:493] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.347429 5921 raft_consensus.cc:515] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.360158 5921 leader_election.cc:290] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.409351 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
W20260620 19:57:23.411676 5766 leader_election.cc:343] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:23.414458 5918 tablet_bootstrap.cc:492] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.414985 5918 ts_tablet_manager.cc:1403] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.100s user 0.054s sys 0.039s
I20260620 19:57:23.415457 5917 raft_consensus.cc:493] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.416042 5917 raft_consensus.cc:515] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.418061 5918 raft_consensus.cc:359] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.418704 5918 raft_consensus.cc:385] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.418586 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" is_pre_election: true
I20260620 19:57:23.419064 5918 raft_consensus.cc:740] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.419839 5918 consensus_queue.cc:260] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
W20260620 19:57:23.420610 5764 leader_election.cc:343] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:23.421096 5764 leader_election.cc:304] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d; no voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:23.421837 5918 ts_tablet_manager.cc:1434] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260620 19:57:23.422806 5918 tablet_bootstrap.cc:492] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.422719 5921 raft_consensus.cc:2749] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:23.426153 5917 leader_election.cc:290] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.428303 5918 tablet_bootstrap.cc:654] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.429785 5916 tablet_bootstrap.cc:492] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.430428 5916 ts_tablet_manager.cc:1403] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.113s user 0.030s sys 0.052s
I20260620 19:57:23.433573 5916 raft_consensus.cc:359] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.434296 5916 raft_consensus.cc:385] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.434630 5916 raft_consensus.cc:740] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.435513 5916 consensus_queue.cc:260] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.440560 5916 ts_tablet_manager.cc:1434] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260620 19:57:23.441509 5916 tablet_bootstrap.cc:492] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.444447 5914 tablet_bootstrap.cc:492] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.446316 5914 ts_tablet_manager.cc:1403] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.115s user 0.045s sys 0.040s
I20260620 19:57:23.448132 5916 tablet_bootstrap.cc:654] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.452381 5914 raft_consensus.cc:359] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.454270 5914 raft_consensus.cc:385] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.456241 5914 raft_consensus.cc:740] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.457468 5914 consensus_queue.cc:260] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.460485 5914 ts_tablet_manager.cc:1434] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260620 19:57:23.461725 5914 tablet_bootstrap.cc:492] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.469858 5914 tablet_bootstrap.cc:654] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.472383 5916 tablet_bootstrap.cc:492] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.472889 5916 ts_tablet_manager.cc:1403] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.004s
I20260620 19:57:23.472990 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
I20260620 19:57:23.473789 5874 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.475274 5691 leader_election.cc:304] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.476343 5917 raft_consensus.cc:2804] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.476713 5917 raft_consensus.cc:493] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.476621 5916 raft_consensus.cc:359] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.477154 5917 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.477389 5916 raft_consensus.cc:385] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.477707 5918 tablet_bootstrap.cc:492] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.477675 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.477736 5916 raft_consensus.cc:740] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.478384 5918 ts_tablet_manager.cc:1403] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.056s user 0.029s sys 0.019s
I20260620 19:57:23.478384 5800 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.479502 5916 consensus_queue.cc:260] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.481371 5937 raft_consensus.cc:493] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.481711 5916 ts_tablet_manager.cc:1434] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 19:57:23.482836 5937 raft_consensus.cc:515] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.484300 5916 tablet_bootstrap.cc:492] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.485000 5937 leader_election.cc:290] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.485724 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "965976254d4a461ba6ad6af4aaa5c441" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.485960 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "965976254d4a461ba6ad6af4aaa5c441" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
I20260620 19:57:23.486287 5800 raft_consensus.cc:2468] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
W20260620 19:57:23.487314 5691 leader_election.cc:343] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:23.487998 5688 leader_election.cc:304] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters: cb9a18ec75324292b1c71c978353bfed
I20260620 19:57:23.489043 5937 raft_consensus.cc:2804] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.489382 5937 raft_consensus.cc:493] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.489856 5937 raft_consensus.cc:3060] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.491673 5916 tablet_bootstrap.cc:654] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.481848 5918 raft_consensus.cc:359] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.494767 5918 raft_consensus.cc:385] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.495178 5918 raft_consensus.cc:740] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.495970 5918 consensus_queue.cc:260] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.498430 5918 ts_tablet_manager.cc:1434] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20260620 19:57:23.498942 5917 raft_consensus.cc:515] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.499570 5918 tablet_bootstrap.cc:492] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.499537 5937 raft_consensus.cc:515] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.504205 5937 leader_election.cc:290] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.505694 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "965976254d4a461ba6ad6af4aaa5c441" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.506026 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "965976254d4a461ba6ad6af4aaa5c441" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:23.506304 5800 raft_consensus.cc:3060] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:57:23.507404 5691 leader_election.cc:343] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:23.509584 5918 tablet_bootstrap.cc:654] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.510265 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:23.510886 5874 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.510913 5917 leader_election.cc:290] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Requested vote from peers cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.511955 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.512550 5799 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.512769 5916 tablet_bootstrap.cc:492] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.513382 5916 ts_tablet_manager.cc:1403] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.004s
I20260620 19:57:23.517246 5916 raft_consensus.cc:359] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.518218 5916 raft_consensus.cc:385] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.518582 5916 raft_consensus.cc:740] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.519748 5916 consensus_queue.cc:260] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.529597 5874 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.531330 5916 ts_tablet_manager.cc:1434] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260620 19:57:23.534296 5916 tablet_bootstrap.cc:492] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.535023 5800 raft_consensus.cc:2468] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.536326 5688 leader_election.cc:304] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters: cb9a18ec75324292b1c71c978353bfed
I20260620 19:57:23.537155 5917 raft_consensus.cc:2804] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.537652 5917 raft_consensus.cc:697] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:23.538441 5917 consensus_queue.cc:237] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.538991 5914 tablet_bootstrap.cc:492] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.539137 5691 leader_election.cc:304] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.539489 5914 ts_tablet_manager.cc:1403] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.078s user 0.024s sys 0.030s
I20260620 19:57:23.540591 5916 tablet_bootstrap.cc:654] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.542102 5937 raft_consensus.cc:2804] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.542575 5937 raft_consensus.cc:697] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:23.542375 5914 raft_consensus.cc:359] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.543102 5914 raft_consensus.cc:385] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.543401 5914 raft_consensus.cc:740] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.543387 5937 consensus_queue.cc:237] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:23.544258 5914 consensus_queue.cc:260] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.536480 5799 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.563288 5918 tablet_bootstrap.cc:492] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.563962 5918 ts_tablet_manager.cc:1403] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.065s user 0.011s sys 0.020s
I20260620 19:57:23.567970 5918 raft_consensus.cc:359] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.569439 5924 raft_consensus.cc:493] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.570078 5924 raft_consensus.cc:515] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.576401 5924 leader_election.cc:290] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.577737 5918 raft_consensus.cc:385] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.578083 5918 raft_consensus.cc:740] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.578887 5918 consensus_queue.cc:260] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
W20260620 19:57:23.580315 5671 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:57:23.581385 5918 ts_tablet_manager.cc:1434] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260620 19:57:23.582314 5918 tablet_bootstrap.cc:492] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.592836 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
W20260620 19:57:23.594161 5837 leader_election.cc:343] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:23.594561 5937 raft_consensus.cc:493] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.595038 5937 raft_consensus.cc:515] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.597340 5937 leader_election.cc:290] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.598098 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
W20260620 19:57:23.599335 5688 leader_election.cc:343] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:57:23.600541 5691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:34813, user_credentials={real_user=slave}} blocked reactor thread for 61506.3us
I20260620 19:57:23.602617 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
I20260620 19:57:23.603170 5874 raft_consensus.cc:2468] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.604720 5947 raft_consensus.cc:493] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.604853 5691 leader_election.cc:304] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters: 994e5d10524b4801a8457e3a58f2527d
I20260620 19:57:23.605201 5947 raft_consensus.cc:515] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.606194 5940 raft_consensus.cc:2804] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.606633 5940 raft_consensus.cc:493] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.607184 5940 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.607882 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.613310 5799 raft_consensus.cc:2468] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.613509 5916 tablet_bootstrap.cc:492] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.613842 5914 ts_tablet_manager.cc:1434] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.074s user 0.022s sys 0.035s
I20260620 19:57:23.613921 5916 ts_tablet_manager.cc:1403] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.080s user 0.019s sys 0.029s
I20260620 19:57:23.614070 5918 tablet_bootstrap.cc:654] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.614404 5688 leader_election.cc:304] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.614930 5914 tablet_bootstrap.cc:492] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.615602 5943 raft_consensus.cc:2804] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.616055 5943 raft_consensus.cc:493] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.616443 5943 raft_consensus.cc:3060] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.611655 5947 leader_election.cc:290] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.612587 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
I20260620 19:57:23.609913 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" is_pre_election: true
I20260620 19:57:23.616977 5916 raft_consensus.cc:359] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.617492 5874 raft_consensus.cc:2468] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.617725 5916 raft_consensus.cc:385] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.618150 5916 raft_consensus.cc:740] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.618995 5916 consensus_queue.cc:260] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.624631 5914 tablet_bootstrap.cc:654] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.627916 5916 ts_tablet_manager.cc:1434] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260620 19:57:23.610901 5623 catalog_manager.cc:5696] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:23.630744 5918 tablet_bootstrap.cc:492] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.631206 5918 ts_tablet_manager.cc:1403] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.049s user 0.025s sys 0.000s
I20260620 19:57:23.631570 5914 tablet_bootstrap.cc:492] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.632037 5914 ts_tablet_manager.cc:1403] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20260620 19:57:23.633981 5918 raft_consensus.cc:359] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.634728 5918 raft_consensus.cc:385] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.634760 5914 raft_consensus.cc:359] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.635214 5916 tablet_bootstrap.cc:492] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.635157 5918 raft_consensus.cc:740] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.635501 5914 raft_consensus.cc:385] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.635869 5914 raft_consensus.cc:740] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.635603 5943 raft_consensus.cc:515] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.636008 5918 consensus_queue.cc:260] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.636785 5914 consensus_queue.cc:260] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.634434 5940 raft_consensus.cc:515] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.640524 5725 raft_consensus.cc:2393] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb9a18ec75324292b1c71c978353bfed in current term 1: Already voted for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in this term.
I20260620 19:57:23.641054 5918 ts_tablet_manager.cc:1434] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260620 19:57:23.639281 5914 ts_tablet_manager.cc:1434] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260620 19:57:23.639475 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.641928 5916 tablet_bootstrap.cc:654] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.642009 5918 tablet_bootstrap.cc:492] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.641983 5799 raft_consensus.cc:3060] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.643937 5838 leader_election.cc:304] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:23.645076 5924 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.648679 5918 tablet_bootstrap.cc:654] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.652801 5799 raft_consensus.cc:2468] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.654381 5914 tablet_bootstrap.cc:492] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.658900 5688 leader_election.cc:304] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.659773 5947 raft_consensus.cc:2804] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.660238 5947 raft_consensus.cc:697] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:23.660418 5918 tablet_bootstrap.cc:492] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.660863 5918 ts_tablet_manager.cc:1403] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20260620 19:57:23.661039 5947 consensus_queue.cc:237] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.663424 5940 leader_election.cc:290] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.663808 5918 raft_consensus.cc:359] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.665918 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:23.667588 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.669860 5924 raft_consensus.cc:2749] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:23.671424 5914 tablet_bootstrap.cc:654] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.671849 5918 raft_consensus.cc:385] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.672271 5799 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.672570 5918 raft_consensus.cc:740] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.672925 5943 leader_election.cc:290] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.673350 5918 consensus_queue.cc:260] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.674220 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:23.674862 5873 raft_consensus.cc:3060] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.675568 5918 ts_tablet_manager.cc:1434] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20260620 19:57:23.676363 5918 tablet_bootstrap.cc:492] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.683029 5918 tablet_bootstrap.cc:654] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.686201 5874 raft_consensus.cc:2468] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.688400 5799 raft_consensus.cc:2468] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.690480 5691 leader_election.cc:304] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.691936 5916 tablet_bootstrap.cc:492] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.692529 5916 ts_tablet_manager.cc:1403] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.058s user 0.018s sys 0.016s
I20260620 19:57:23.694180 5873 raft_consensus.cc:2468] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.695827 5916 raft_consensus.cc:359] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.696590 5916 raft_consensus.cc:385] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.696875 5916 raft_consensus.cc:740] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.697760 5943 raft_consensus.cc:2804] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.697677 5916 consensus_queue.cc:260] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.698264 5943 raft_consensus.cc:697] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:23.699090 5943 consensus_queue.cc:237] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.703011 5916 ts_tablet_manager.cc:1434] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 19:57:23.704028 5916 tablet_bootstrap.cc:492] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.710417 5916 tablet_bootstrap.cc:654] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.721262 5914 tablet_bootstrap.cc:492] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.721976 5914 ts_tablet_manager.cc:1403] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.068s user 0.019s sys 0.006s
I20260620 19:57:23.723671 5622 catalog_manager.cc:5696] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:23.726117 5622 catalog_manager.cc:5696] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:23.728457 5914 raft_consensus.cc:359] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.729561 5914 raft_consensus.cc:385] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.728581 5622 catalog_manager.cc:5696] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:23.730090 5914 raft_consensus.cc:740] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.731328 5914 consensus_queue.cc:260] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.734329 5924 raft_consensus.cc:493] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.735046 5924 raft_consensus.cc:515] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.735409 5914 ts_tablet_manager.cc:1434] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20260620 19:57:23.736399 5914 tablet_bootstrap.cc:492] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.737131 5918 tablet_bootstrap.cc:492] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.737196 5924 leader_election.cc:290] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.737769 5918 ts_tablet_manager.cc:1403] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.048s
I20260620 19:57:23.739938 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" is_pre_election: true
I20260620 19:57:23.740576 5725 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:23.740950 5918 raft_consensus.cc:359] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.741794 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.742470 5914 tablet_bootstrap.cc:654] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.743304 5838 leader_election.cc:304] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.744216 5924 raft_consensus.cc:2804] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.744596 5924 raft_consensus.cc:493] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.745314 5924 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.742410 5799 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:23.753477 5914 tablet_bootstrap.cc:492] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.753919 5914 ts_tablet_manager.cc:1403] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20260620 19:57:23.755460 5924 raft_consensus.cc:515] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.756436 5914 raft_consensus.cc:359] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.757091 5914 raft_consensus.cc:385] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.757352 5914 raft_consensus.cc:740] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.758330 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:23.758199 5914 consensus_queue.cc:260] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.758872 5725 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.760433 5914 ts_tablet_manager.cc:1434] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:57:23.761346 5914 tablet_bootstrap.cc:492] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.766688 5914 tablet_bootstrap.cc:654] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.766520 5725 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 1.
I20260620 19:57:23.767859 5838 leader_election.cc:304] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.775700 5961 raft_consensus.cc:2804] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.775627 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.776412 5916 tablet_bootstrap.cc:492] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.776551 5799 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.777045 5916 ts_tablet_manager.cc:1403] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.073s user 0.018s sys 0.049s
I20260620 19:57:23.778999 5961 raft_consensus.cc:697] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Becoming Leader. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:23.780110 5961 consensus_queue.cc:237] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.783437 5916 raft_consensus.cc:359] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.784143 5916 raft_consensus.cc:385] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.784425 5914 tablet_bootstrap.cc:492] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.784456 5916 raft_consensus.cc:740] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.784998 5914 ts_tablet_manager.cc:1403] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.009s
I20260620 19:57:23.785774 5799 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 1.
I20260620 19:57:23.785660 5916 consensus_queue.cc:260] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.786320 5924 leader_election.cc:290] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.787452 5914 raft_consensus.cc:359] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.788085 5914 raft_consensus.cc:385] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.741804 5918 raft_consensus.cc:385] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.788347 5914 raft_consensus.cc:740] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.788885 5918 raft_consensus.cc:740] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.789227 5914 consensus_queue.cc:260] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.789711 5918 consensus_queue.cc:260] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.791301 5914 ts_tablet_manager.cc:1434] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:57:23.792129 5914 tablet_bootstrap.cc:492] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98: Bootstrap starting.
I20260620 19:57:23.794111 5918 ts_tablet_manager.cc:1434] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.056s user 0.008s sys 0.000s
I20260620 19:57:23.795059 5918 tablet_bootstrap.cc:492] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.800325 5918 tablet_bootstrap.cc:654] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.800325 5914 tablet_bootstrap.cc:654] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.808794 5916 ts_tablet_manager.cc:1434] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.031s user 0.019s sys 0.011s
I20260620 19:57:23.809842 5916 tablet_bootstrap.cc:492] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.810448 5914 tablet_bootstrap.cc:492] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98: No bootstrap required, opened a new log
I20260620 19:57:23.810906 5914 ts_tablet_manager.cc:1403] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.006s
I20260620 19:57:23.811647 5918 tablet_bootstrap.cc:492] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.812109 5918 ts_tablet_manager.cc:1403] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20260620 19:57:23.813345 5914 raft_consensus.cc:359] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.814105 5914 raft_consensus.cc:385] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.814364 5914 raft_consensus.cc:740] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.814858 5918 raft_consensus.cc:359] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.815111 5914 consensus_queue.cc:260] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.815577 5918 raft_consensus.cc:385] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.815941 5918 raft_consensus.cc:740] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.816215 5916 tablet_bootstrap.cc:654] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.816998 5918 consensus_queue.cc:260] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.819113 5914 ts_tablet_manager.cc:1434] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 19:57:23.819211 5918 ts_tablet_manager.cc:1434] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:57:23.820425 5918 tablet_bootstrap.cc:492] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d: Bootstrap starting.
I20260620 19:57:23.822116 5622 catalog_manager.cc:5696] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed reported cstate change: term changed from 0 to 1, leader changed from <none> to cb9a18ec75324292b1c71c978353bfed (127.4.244.131). New cstate: current_term: 1 leader_uuid: "cb9a18ec75324292b1c71c978353bfed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:23.830005 5918 tablet_bootstrap.cc:654] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.835618 5918 tablet_bootstrap.cc:492] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d: No bootstrap required, opened a new log
I20260620 19:57:23.835896 5916 tablet_bootstrap.cc:492] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.836017 5918 ts_tablet_manager.cc:1403] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20260620 19:57:23.836342 5916 ts_tablet_manager.cc:1403] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.000s
I20260620 19:57:23.838141 5918 raft_consensus.cc:359] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.838733 5918 raft_consensus.cc:385] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.838999 5918 raft_consensus.cc:740] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.838980 5916 raft_consensus.cc:359] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.839648 5916 raft_consensus.cc:385] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.839917 5916 raft_consensus.cc:740] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.840018 5918 consensus_queue.cc:260] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.840554 5916 consensus_queue.cc:260] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.841483 5918 ts_tablet_manager.cc:1434] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:57:23.843472 5916 ts_tablet_manager.cc:1434] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:23.844328 5916 tablet_bootstrap.cc:492] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.849283 5916 tablet_bootstrap.cc:654] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.854306 5916 tablet_bootstrap.cc:492] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.854698 5916 ts_tablet_manager.cc:1403] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 19:57:23.854756 5961 raft_consensus.cc:493] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.855177 5943 raft_consensus.cc:493] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.855239 5961 raft_consensus.cc:515] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.855588 5943 raft_consensus.cc:515] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.856259 5940 raft_consensus.cc:493] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.856691 5940 raft_consensus.cc:515] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.857342 5961 leader_election.cc:290] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.857496 5943 leader_election.cc:290] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.857280 5916 raft_consensus.cc:359] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.857969 5916 raft_consensus.cc:385] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.858285 5916 raft_consensus.cc:740] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.858552 5940 leader_election.cc:290] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.858914 5916 consensus_queue.cc:260] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.859681 5799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.859812 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.860237 5799 raft_consensus.cc:2468] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:23.860400 5798 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.860817 5916 ts_tablet_manager.cc:1434] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:57:23.861653 5688 leader_election.cc:304] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.861944 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.861982 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
I20260620 19:57:23.862421 5940 raft_consensus.cc:2804] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.862725 5874 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.862555 5800 raft_consensus.cc:2468] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.862975 5940 raft_consensus.cc:493] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.861615 5837 leader_election.cc:304] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:23.863900 5940 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.863257 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" is_pre_election: true
I20260620 19:57:23.863994 5688 leader_election.cc:304] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.864590 5725 raft_consensus.cc:2468] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:23.862653 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
I20260620 19:57:23.865943 5943 raft_consensus.cc:2804] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.866321 5943 raft_consensus.cc:493] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.861631 5916 tablet_bootstrap.cc:492] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed: Bootstrap starting.
I20260620 19:57:23.866843 5943 raft_consensus.cc:3060] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.866300 5873 raft_consensus.cc:2468] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:23.870711 5961 raft_consensus.cc:2804] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.871165 5940 raft_consensus.cc:515] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.872745 5916 tablet_bootstrap.cc:654] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:23.873888 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.874202 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:23.871567 5961 raft_consensus.cc:493] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.874447 5800 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.874810 5873 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.876768 5940 leader_election.cc:290] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.877434 5961 raft_consensus.cc:3060] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.879004 5916 tablet_bootstrap.cc:492] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed: No bootstrap required, opened a new log
I20260620 19:57:23.879483 5916 ts_tablet_manager.cc:1403] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20260620 19:57:23.881208 5800 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.882102 5873 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.882309 5688 leader_election.cc:304] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.882107 5916 raft_consensus.cc:359] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.883100 5916 raft_consensus.cc:385] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:23.883299 5940 raft_consensus.cc:2804] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.883416 5916 raft_consensus.cc:740] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Initialized, Role: FOLLOWER
I20260620 19:57:23.883947 5940 raft_consensus.cc:697] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:23.884322 5961 raft_consensus.cc:515] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.884207 5916 consensus_queue.cc:260] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.884856 5940 consensus_queue.cc:237] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.886250 5916 ts_tablet_manager.cc:1434] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260620 19:57:23.887010 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:23.887854 5965 raft_consensus.cc:493] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:23.888278 5965 raft_consensus.cc:515] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.888697 5943 raft_consensus.cc:515] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.889832 5725 raft_consensus.cc:2393] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cb9a18ec75324292b1c71c978353bfed in current term 1: Already voted for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in this term.
I20260620 19:57:23.890494 5965 leader_election.cc:290] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.891651 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa96289638f14c039a9f769c43a5d912" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:23.892262 5800 raft_consensus.cc:2468] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:23.892349 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa96289638f14c039a9f769c43a5d912" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" is_pre_election: true
I20260620 19:57:23.893028 5724 raft_consensus.cc:2468] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:23.893347 5837 leader_election.cc:304] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:23.894104 5965 raft_consensus.cc:2804] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:23.894464 5965 raft_consensus.cc:493] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:23.894855 5965 raft_consensus.cc:3060] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.896363 5943 leader_election.cc:290] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:23.897281 5961 leader_election.cc:290] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.897706 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:23.898419 5873 raft_consensus.cc:2393] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dcc4c58fa6cb49a9a74281bb8e748e98 in current term 1: Already voted for candidate cb9a18ec75324292b1c71c978353bfed in this term.
I20260620 19:57:23.899593 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.900194 5800 raft_consensus.cc:3060] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.899703 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90568e63c88430ea023f53fb89ac8d4" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.902110 5837 leader_election.cc:304] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:23.903366 5965 raft_consensus.cc:515] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.905344 5965 leader_election.cc:290] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:23.905381 5800 raft_consensus.cc:2468] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:23.905921 5965 raft_consensus.cc:2749] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:23.906368 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa96289638f14c039a9f769c43a5d912" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:23.906982 5798 raft_consensus.cc:3060] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.906950 5688 leader_election.cc:304] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters: cb9a18ec75324292b1c71c978353bfed
I20260620 19:57:23.907869 5940 raft_consensus.cc:2804] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.908511 5940 raft_consensus.cc:697] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:23.908785 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa96289638f14c039a9f769c43a5d912" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:23.907961 5621 catalog_manager.cc:5696] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:23.909379 5724 raft_consensus.cc:3060] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:23.909538 5940 consensus_queue.cc:237] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.916188 5724 raft_consensus.cc:2468] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 1.
I20260620 19:57:23.917196 5838 leader_election.cc:304] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:23.918478 5961 raft_consensus.cc:2804] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:23.918051 5798 raft_consensus.cc:2468] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 1.
I20260620 19:57:23.919371 5961 raft_consensus.cc:697] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Becoming Leader. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:23.920228 5961 consensus_queue.cc:237] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:23.927591 5621 catalog_manager.cc:5696] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed reported cstate change: term changed from 0 to 1, leader changed from <none> to cb9a18ec75324292b1c71c978353bfed (127.4.244.131). New cstate: current_term: 1 leader_uuid: "cb9a18ec75324292b1c71c978353bfed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:23.927650 5622 catalog_manager.cc:5696] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:23.988126 5940 consensus_queue.cc:1048] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:23.996922 5873 raft_consensus.cc:3060] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:24.005237 5943 consensus_queue.cc:1048] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.015070 5943 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.025321 5940 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.103569 5940 consensus_queue.cc:1048] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.113441 5940 consensus_queue.cc:1048] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.141160 5943 consensus_queue.cc:1048] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.145471 5943 raft_consensus.cc:493] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:24.145874 5943 raft_consensus.cc:515] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.148800 5943 leader_election.cc:290] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:24.149544 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed" is_pre_election: true
I20260620 19:57:24.150166 5873 raft_consensus.cc:2468] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:24.151280 5691 leader_election.cc:304] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:24.151657 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:24.152202 5798 raft_consensus.cc:2468] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 0.
I20260620 19:57:24.152513 5947 raft_consensus.cc:2804] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:24.152859 5947 raft_consensus.cc:493] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:24.153211 5947 raft_consensus.cc:3060] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:24.153258 5917 consensus_queue.cc:1048] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.153649 5961 raft_consensus.cc:493] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:24.154136 5961 raft_consensus.cc:515] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.156109 5961 leader_election.cc:290] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:24.158727 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" is_pre_election: true
I20260620 19:57:24.159315 5724 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:24.159912 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d" is_pre_election: true
I20260620 19:57:24.160528 5798 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 0.
I20260620 19:57:24.160748 5838 leader_election.cc:304] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:24.162674 5965 raft_consensus.cc:2804] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:24.163045 5965 raft_consensus.cc:493] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:24.163374 5965 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:24.160678 5947 raft_consensus.cc:515] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.168208 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:24.168605 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:24.168735 5798 raft_consensus.cc:3060] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:24.170006 5965 raft_consensus.cc:515] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.170912 5874 raft_consensus.cc:3060] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:24.172154 5947 leader_election.cc:290] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:24.173427 5965 leader_election.cc:290] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:24.173550 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:24.174136 5724 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:24.175544 5798 raft_consensus.cc:2468] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:24.178478 5688 leader_election.cc:304] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:24.179045 5874 raft_consensus.cc:2468] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 1.
I20260620 19:57:24.179237 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:24.179504 5947 raft_consensus.cc:2804] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:24.179857 5798 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:24.180193 5947 raft_consensus.cc:697] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:24.180547 5724 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 1.
I20260620 19:57:24.181061 5947 consensus_queue.cc:237] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.181928 5838 leader_election.cc:304] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:24.182794 5965 raft_consensus.cc:2804] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:24.183940 5965 raft_consensus.cc:697] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Becoming Leader. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:24.184754 5965 consensus_queue.cc:237] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.186295 5798 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 1.
I20260620 19:57:24.192252 5622 catalog_manager.cc:5696] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:24.194983 5621 catalog_manager.cc:5696] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed reported cstate change: term changed from 0 to 1, leader changed from <none> to cb9a18ec75324292b1c71c978353bfed (127.4.244.131). New cstate: current_term: 1 leader_uuid: "cb9a18ec75324292b1c71c978353bfed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:24.203202 5965 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.213595 5965 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.293409 5947 consensus_queue.cc:1048] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.294286 5943 consensus_queue.cc:1048] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.309101 5947 consensus_queue.cc:1048] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.312198 5917 consensus_queue.cc:1048] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.350479 5965 consensus_queue.cc:1048] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.359623 5961 consensus_queue.cc:1048] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.470844 5873 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7123643714146939d0d82713a70121e"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
mode: GRACEFUL
new_leader_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:47028
I20260620 19:57:24.471417 5873 raft_consensus.cc:606] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Received request to transfer leadership to TS 994e5d10524b4801a8457e3a58f2527d
I20260620 19:57:24.483497 5724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49542
I20260620 19:57:24.483964 5724 raft_consensus.cc:606] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Received request to transfer leadership to TS 994e5d10524b4801a8457e3a58f2527d
I20260620 19:57:24.488608 5873 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
mode: GRACEFUL
new_leader_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:47028
I20260620 19:57:24.489087 5873 raft_consensus.cc:606] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Received request to transfer leadership to TS 994e5d10524b4801a8457e3a58f2527d
I20260620 19:57:24.493796 5724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49542
I20260620 19:57:24.494259 5724 raft_consensus.cc:606] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Received request to transfer leadership to TS 994e5d10524b4801a8457e3a58f2527d
I20260620 19:57:24.581123 5947 consensus_queue.cc:1048] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.591776 5947 consensus_queue.cc:1048] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.604485 5917 raft_consensus.cc:993] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.604995 5947 raft_consensus.cc:1081] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Signalling peer 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.606379 5798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49402
I20260620 19:57:24.606902 5798 raft_consensus.cc:493] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:24.607206 5798 raft_consensus.cc:3060] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.611601 5798 raft_consensus.cc:515] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.613044 5798 leader_election.cc:290] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:24.613848 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:24.614002 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9112ebb71314faba51ce60093f7694f" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:24.614494 5724 raft_consensus.cc:3055] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:24.614578 5873 raft_consensus.cc:3060] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.614926 5724 raft_consensus.cc:740] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:24.615716 5724 consensus_queue.cc:260] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.616946 5724 raft_consensus.cc:3060] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.619522 5873 raft_consensus.cc:2468] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.620491 5766 leader_election.cc:304] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:24.621232 5933 raft_consensus.cc:2804] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:24.622233 5724 raft_consensus.cc:2468] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.622428 5933 raft_consensus.cc:697] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Leader. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:24.623323 5933 consensus_queue.cc:237] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.628814 5623 catalog_manager.cc:5696] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d reported cstate change: term changed from 1 to 2, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to 994e5d10524b4801a8457e3a58f2527d (127.4.244.130). New cstate: current_term: 2 leader_uuid: "994e5d10524b4801a8457e3a58f2527d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:24.733614 5924 raft_consensus.cc:993] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed: : Instructing follower 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.734086 5961 raft_consensus.cc:1081] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Signalling peer 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.735507 5798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7123643714146939d0d82713a70121e"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49408
I20260620 19:57:24.735963 5798 raft_consensus.cc:493] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:24.736274 5798 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.740832 5798 raft_consensus.cc:515] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.742386 5798 leader_election.cc:290] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:24.743113 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:24.743353 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:24.743595 5724 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.743902 5873 raft_consensus.cc:3055] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:24.744189 5873 raft_consensus.cc:740] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:24.744830 5873 consensus_queue.cc:260] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.745673 5873 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.747947 5724 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.748849 5764 leader_election.cc:304] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:24.749567 5933 raft_consensus.cc:2804] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:24.749939 5933 raft_consensus.cc:697] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Leader. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:24.750393 5873 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.750695 5933 consensus_queue.cc:237] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.757659 5622 catalog_manager.cc:5696] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d reported cstate change: term changed from 1 to 2, leader changed from cb9a18ec75324292b1c71c978353bfed (127.4.244.131) to 994e5d10524b4801a8457e3a58f2527d (127.4.244.130). New cstate: current_term: 2 leader_uuid: "994e5d10524b4801a8457e3a58f2527d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:24.786329 5961 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.796567 5961 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:24.803804 5963 raft_consensus.cc:993] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed: : Instructing follower 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.804265 5961 raft_consensus.cc:1081] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Signalling peer 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.805773 5798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49408
I20260620 19:57:24.806301 5798 raft_consensus.cc:493] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:24.806720 5798 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.811121 5798 raft_consensus.cc:515] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.812920 5798 leader_election.cc:290] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:24.813581 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:24.813963 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:24.814184 5724 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.814579 5873 raft_consensus.cc:3055] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:24.814914 5873 raft_consensus.cc:740] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:24.815623 5873 consensus_queue.cc:260] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.816478 5873 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.819564 5724 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.820423 5764 leader_election.cc:304] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:24.821033 5933 raft_consensus.cc:2804] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:24.821228 5873 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.821393 5933 raft_consensus.cc:697] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Leader. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:24.822152 5933 consensus_queue.cc:237] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.827445 5622 catalog_manager.cc:5696] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d reported cstate change: term changed from 1 to 2, leader changed from cb9a18ec75324292b1c71c978353bfed (127.4.244.131) to 994e5d10524b4801a8457e3a58f2527d (127.4.244.130). New cstate: current_term: 2 leader_uuid: "994e5d10524b4801a8457e3a58f2527d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:24.869076 5947 raft_consensus.cc:993] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.869529 5943 raft_consensus.cc:1081] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Signalling peer 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:24.870970 5798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49402
I20260620 19:57:24.871424 5798 raft_consensus.cc:493] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:24.871773 5798 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.876206 5798 raft_consensus.cc:515] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.877874 5798 leader_election.cc:290] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:24.878563 5724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:24.878772 5873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:24.879220 5724 raft_consensus.cc:3055] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:24.879287 5873 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.879689 5724 raft_consensus.cc:740] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:24.880350 5724 consensus_queue.cc:260] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.881183 5724 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:24.883741 5873 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.884649 5766 leader_election.cc:304] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:24.885376 5933 raft_consensus.cc:2804] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:24.885746 5933 raft_consensus.cc:697] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Leader. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:24.885862 5724 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:24.886466 5933 consensus_queue.cc:237] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:24.891595 5622 catalog_manager.cc:5696] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d reported cstate change: term changed from 1 to 2, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to 994e5d10524b4801a8457e3a58f2527d (127.4.244.130). New cstate: current_term: 2 leader_uuid: "994e5d10524b4801a8457e3a58f2527d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:25.079720 5874 raft_consensus.cc:1275] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.080960 5933 consensus_queue.cc:1048] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:25.087987 5724 raft_consensus.cc:1275] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.089416 5933 consensus_queue.cc:1048] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:25.142820 5724 raft_consensus.cc:1275] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.144001 5933 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:25.150957 5874 raft_consensus.cc:1275] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.152669 6001 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:25.262099 5874 raft_consensus.cc:1275] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.263243 6001 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:25.270550 5724 raft_consensus.cc:1275] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.272413 6001 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:25.276996 5725 raft_consensus.cc:1275] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.278416 6001 consensus_queue.cc:1048] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:25.285784 5874 raft_consensus.cc:1275] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:25.287573 5933 consensus_queue.cc:1048] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:27.525559 5074 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20260620 19:57:27.526270 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:27.546208 5725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5271c718decc4e10895728893bf42fcf"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:54838
I20260620 19:57:27.546728 5725 raft_consensus.cc:606] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Received request to transfer leadership to TS cb9a18ec75324292b1c71c978353bfed
I20260620 19:57:27.548936 5725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:54838
I20260620 19:57:27.549320 5725 raft_consensus.cc:606] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Received request to transfer leadership to TS cb9a18ec75324292b1c71c978353bfed
I20260620 19:57:27.550262 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:57:27.550725 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:27.657940 5943 raft_consensus.cc:993] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:27.658357 6009 raft_consensus.cc:1081] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Signalling peer cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:27.659878 5873 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5271c718decc4e10895728893bf42fcf"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:47012
I20260620 19:57:27.660415 5873 raft_consensus.cc:493] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:27.660753 5873 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:27.665182 5873 raft_consensus.cc:515] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:27.666764 5873 leader_election.cc:290] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:27.667637 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:27.667827 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:27.668298 5725 raft_consensus.cc:3055] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:27.668438 5798 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:27.668645 5725 raft_consensus.cc:740] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:27.669477 5725 consensus_queue.cc:260] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:27.670518 5725 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:27.672881 5798 raft_consensus.cc:2468] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 2.
I20260620 19:57:27.673836 5837 leader_election.cc:304] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:27.674496 5961 raft_consensus.cc:2804] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:27.675469 5961 raft_consensus.cc:697] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Becoming Leader. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:27.675949 5725 raft_consensus.cc:2468] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 2.
I20260620 19:57:27.676369 5961 consensus_queue.cc:237] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:27.683666 5623 catalog_manager.cc:5696] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed reported cstate change: term changed from 1 to 2, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to cb9a18ec75324292b1c71c978353bfed (127.4.244.131). New cstate: current_term: 2 leader_uuid: "cb9a18ec75324292b1c71c978353bfed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:27.922933 6009 raft_consensus.cc:993] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:27.923326 5943 raft_consensus.cc:1081] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Signalling peer cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:27.924664 5873 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:47012
I20260620 19:57:27.925161 5873 raft_consensus.cc:493] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:27.925453 5873 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:27.929850 5873 raft_consensus.cc:515] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:27.931329 5873 leader_election.cc:290] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:27.932116 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:27.932389 5798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:27.932598 5725 raft_consensus.cc:3055] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:27.932891 5798 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:27.932911 5725 raft_consensus.cc:740] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:27.933662 5725 consensus_queue.cc:260] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:27.934487 5725 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:27.936911 5798 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 2.
I20260620 19:57:27.937767 5837 leader_election.cc:304] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:27.938438 5961 raft_consensus.cc:2804] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:27.938925 5961 raft_consensus.cc:697] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Becoming Leader. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:27.939256 5725 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 2.
I20260620 19:57:27.939723 5961 consensus_queue.cc:237] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:27.947496 5623 catalog_manager.cc:5696] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed reported cstate change: term changed from 1 to 2, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to cb9a18ec75324292b1c71c978353bfed (127.4.244.131). New cstate: current_term: 2 leader_uuid: "cb9a18ec75324292b1c71c978353bfed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:28.077934 5725 raft_consensus.cc:1275] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:28.079208 5961 consensus_queue.cc:1048] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:28.083168 5798 raft_consensus.cc:1275] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:28.084306 5961 consensus_queue.cc:1048] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:28.347527 5798 raft_consensus.cc:1275] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:28.348732 6019 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:28.359282 5725 raft_consensus.cc:1275] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:28.360713 6025 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:28.551533 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:28.556777 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:28.557175 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:29.558137 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:29.563441 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:29.563862 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:30.564704 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:30.570279 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:30.570744 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:31.572016 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:31.581084 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:31.581678 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:32.582937 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:32.588060 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:32.588567 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:33.589504 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:33.594856 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:33.595240 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:34.596068 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:34.601105 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:34.601577 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:35.602473 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:35.608384 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:35.608743 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:36.609643 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:36.615417 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:36.615790 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:37.644490 5074 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260620 19:57:37.649148 5874 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
mode: GRACEFUL
new_leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:47028
I20260620 19:57:37.649834 5874 raft_consensus.cc:606] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Received request to transfer leadership to TS dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:37.657050 5874 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5271c718decc4e10895728893bf42fcf"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
mode: GRACEFUL
new_leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:47028
I20260620 19:57:37.657513 5874 raft_consensus.cc:606] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Received request to transfer leadership to TS dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:37.680764 5800 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7123643714146939d0d82713a70121e"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
mode: GRACEFUL
new_leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:52320
I20260620 19:57:37.681275 5800 raft_consensus.cc:606] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Received request to transfer leadership to TS dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:37.685851 5800 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
mode: GRACEFUL
new_leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:52320
I20260620 19:57:37.686318 5800 raft_consensus.cc:606] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Received request to transfer leadership to TS dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:37.691015 5800 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
mode: GRACEFUL
new_leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:52320
I20260620 19:57:37.691509 5800 raft_consensus.cc:606] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Received request to transfer leadership to TS dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:37.744598 6034 raft_consensus.cc:993] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed: : Instructing follower dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.745004 6034 raft_consensus.cc:1081] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Signalling peer dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.746382 5725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:49534
I20260620 19:57:37.746847 5725 raft_consensus.cc:493] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:37.747102 5725 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.751879 5725 raft_consensus.cc:515] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:37.753443 5725 leader_election.cc:290] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Requested vote from peers cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:37.754406 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:37.754488 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:37.755128 5874 raft_consensus.cc:3055] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:37.755167 5800 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.755441 5874 raft_consensus.cc:740] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:37.756174 5874 consensus_queue.cc:260] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:37.757226 5874 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.760223 5800 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:37.761444 5688 leader_election.cc:304] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:37.762234 6033 raft_consensus.cc:2804] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:37.762238 5874 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:37.763496 6033 raft_consensus.cc:697] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:37.764371 6033 consensus_queue.cc:237] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:37.771557 5621 catalog_manager.cc:5696] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 2 to 3, leader changed from cb9a18ec75324292b1c71c978353bfed (127.4.244.131) to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:37.902297 6017 raft_consensus.cc:993] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d: : Instructing follower dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.902812 6045 raft_consensus.cc:1081] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Signalling peer dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.904099 5725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:49526
I20260620 19:57:37.904556 5725 raft_consensus.cc:493] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:37.904848 5725 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.908995 5725 raft_consensus.cc:515] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:37.910310 5725 leader_election.cc:290] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:37.911064 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:37.911368 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:37.911540 5800 raft_consensus.cc:3055] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:37.911888 5874 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.911893 5800 raft_consensus.cc:740] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:37.912773 5800 consensus_queue.cc:260] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:37.913625 5800 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.916719 5874 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:37.917496 5800 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:37.917856 5691 leader_election.cc:304] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:37.918838 6033 raft_consensus.cc:2804] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:37.919276 6033 raft_consensus.cc:697] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:37.919997 6033 consensus_queue.cc:237] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:37.926265 5621 catalog_manager.cc:5696] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 2 to 3, leader changed from 994e5d10524b4801a8457e3a58f2527d (127.4.244.130) to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:37.942088 6017 raft_consensus.cc:993] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d: : Instructing follower dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.942487 6045 raft_consensus.cc:1081] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Signalling peer dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.943724 5725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7123643714146939d0d82713a70121e"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:49526
I20260620 19:57:37.944078 5725 raft_consensus.cc:493] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:37.944288 5725 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.948230 5725 raft_consensus.cc:515] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:37.949468 5725 leader_election.cc:290] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:37.950169 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:37.950547 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:37.950721 5800 raft_consensus.cc:3055] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:37.950992 5800 raft_consensus.cc:740] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:37.951043 5874 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.951640 5800 consensus_queue.cc:260] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:37.952409 5800 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:37.955475 5874 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:37.956318 5800 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:37.956348 5691 leader_election.cc:304] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:37.957182 6033 raft_consensus.cc:2804] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:37.957633 6033 raft_consensus.cc:697] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:37.958400 6033 consensus_queue.cc:237] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:37.964169 5621 catalog_manager.cc:5696] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 2 to 3, leader changed from 994e5d10524b4801a8457e3a58f2527d (127.4.244.130) to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:37.993652 6017 raft_consensus.cc:993] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d: : Instructing follower dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.994102 6045 raft_consensus.cc:1081] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Signalling peer dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:37.995436 5725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:49526
I20260620 19:57:37.995873 5725 raft_consensus.cc:493] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:37.996160 5725 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:38.000165 5725 raft_consensus.cc:515] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:38.001420 5725 leader_election.cc:290] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:38.002020 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:38.002506 5800 raft_consensus.cc:3055] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:38.002440 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1814c9aa9f640b7bc73abee51e147c3" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:38.002868 5800 raft_consensus.cc:740] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:38.003026 5874 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:38.003571 5800 consensus_queue.cc:260] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:38.004371 5800 raft_consensus.cc:3060] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:38.007280 5874 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:38.008152 5800 raft_consensus.cc:2468] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:38.008275 5691 leader_election.cc:304] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:38.009094 6033 raft_consensus.cc:2804] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:38.009472 6033 raft_consensus.cc:697] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:38.010193 6033 consensus_queue.cc:237] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:38.015987 5621 catalog_manager.cc:5696] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 2 to 3, leader changed from 994e5d10524b4801a8457e3a58f2527d (127.4.244.130) to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:38.050853 6038 raft_consensus.cc:993] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed: : Instructing follower dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:38.051321 6034 raft_consensus.cc:1081] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Signalling peer dcc4c58fa6cb49a9a74281bb8e748e98 to start an election
I20260620 19:57:38.052651 5725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5271c718decc4e10895728893bf42fcf"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
from {username='slave'} at 127.0.0.1:49534
I20260620 19:57:38.053139 5725 raft_consensus.cc:493] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:38.053398 5725 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:38.057386 5725 raft_consensus.cc:515] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:38.058665 5725 leader_election.cc:290] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Requested vote from peers 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:38.059393 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:38.059685 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5271c718decc4e10895728893bf42fcf" candidate_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:38.059886 5800 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:38.060218 5874 raft_consensus.cc:3055] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:57:38.060496 5874 raft_consensus.cc:740] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:38.061120 5874 consensus_queue.cc:260] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:38.061980 5874 raft_consensus.cc:3060] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:38.064872 5800 raft_consensus.cc:2468] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:38.065987 5688 leader_election.cc:304] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98; no voters:
I20260620 19:57:38.066789 6033 raft_consensus.cc:2804] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:38.068171 6033 raft_consensus.cc:697] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Leader. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:38.068369 5874 raft_consensus.cc:2468] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc4c58fa6cb49a9a74281bb8e748e98 in term 3.
I20260620 19:57:38.069024 6033 consensus_queue.cc:237] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:38.075606 5621 catalog_manager.cc:5696] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 reported cstate change: term changed from 2 to 3, leader changed from cb9a18ec75324292b1c71c978353bfed (127.4.244.131) to dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:38.296124 5874 raft_consensus.cc:1275] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.297353 6033 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:38.307680 5800 raft_consensus.cc:1275] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.308871 6033 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:38.418021 5800 raft_consensus.cc:1275] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.419113 6033 consensus_queue.cc:1048] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:38.427520 5874 raft_consensus.cc:1275] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.430258 6057 consensus_queue.cc:1048] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:57:38.444211 5800 raft_consensus.cc:1275] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.445284 6057 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:38.453147 5874 raft_consensus.cc:1275] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.455324 6049 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:57:38.478767 5873 raft_consensus.cc:1275] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.479375 5800 raft_consensus.cc:1275] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.479956 6049 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:38.480520 6057 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:38.629626 5873 raft_consensus.cc:1275] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.630815 6057 consensus_queue.cc:1048] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:38.638779 5800 raft_consensus.cc:1275] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Refusing update from remote peer dcc4c58fa6cb49a9a74281bb8e748e98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:57:38.640475 6039 consensus_queue.cc:1048] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:57:40.725991 5074 auto_leader_rebalancer-test.cc:399] The leader distribution is
I20260620 19:57:40.726631 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:40.733223 5725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:54838
I20260620 19:57:40.733676 5725 raft_consensus.cc:606] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Received request to transfer leadership to TS 994e5d10524b4801a8457e3a58f2527d
I20260620 19:57:40.735975 5725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:54838
I20260620 19:57:40.736384 5725 raft_consensus.cc:606] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Received request to transfer leadership to TS cb9a18ec75324292b1c71c978353bfed
I20260620 19:57:40.738451 5725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7123643714146939d0d82713a70121e"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:54838
I20260620 19:57:40.738888 5725 raft_consensus.cc:606] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Received request to transfer leadership to TS 994e5d10524b4801a8457e3a58f2527d
I20260620 19:57:40.740826 5725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb"
dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
mode: GRACEFUL
new_leader_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:54838
I20260620 19:57:40.741217 5725 raft_consensus.cc:606] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Received request to transfer leadership to TS cb9a18ec75324292b1c71c978353bfed
I20260620 19:57:40.742085 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260620 19:57:40.742478 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:40.800318 6066 raft_consensus.cc:993] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:40.800797 6033 raft_consensus.cc:1081] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Signalling peer 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:40.802073 5800 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49402
I20260620 19:57:40.802436 5800 raft_consensus.cc:493] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:40.802729 5800 raft_consensus.cc:3060] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:40.806629 5800 raft_consensus.cc:515] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:40.808286 5800 leader_election.cc:290] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:40.809130 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:40.809178 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16dfb3ea9ba94e66a939bfea5adb58ff" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:40.809752 5725 raft_consensus.cc:3055] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:40.809782 5874 raft_consensus.cc:3060] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:40.810098 5725 raft_consensus.cc:740] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:40.810760 5725 consensus_queue.cc:260] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:40.811702 5725 raft_consensus.cc:3060] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:40.813828 5874 raft_consensus.cc:2468] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:40.814877 5766 leader_election.cc:304] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:40.815487 6045 raft_consensus.cc:2804] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:40.815762 5725 raft_consensus.cc:2468] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 2.
I20260620 19:57:40.816416 6045 raft_consensus.cc:697] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Becoming Leader. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:40.817296 6045 consensus_queue.cc:237] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:40.823153 5623 catalog_manager.cc:5696] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d reported cstate change: term changed from 1 to 2, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to 994e5d10524b4801a8457e3a58f2527d (127.4.244.130). New cstate: current_term: 2 leader_uuid: "994e5d10524b4801a8457e3a58f2527d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:40.905061 6066 raft_consensus.cc:993] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:40.905496 6033 raft_consensus.cc:1081] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Signalling peer cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:40.906754 5874 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:47012
I20260620 19:57:40.907173 5874 raft_consensus.cc:493] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:40.907440 5874 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:40.911610 5874 raft_consensus.cc:515] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:40.913358 5874 leader_election.cc:290] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:40.914259 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:40.914343 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06c815cc3e64a9fb26c529634a70ccb" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:40.914903 5725 raft_consensus.cc:3055] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:57:40.914952 5800 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:40.915192 5725 raft_consensus.cc:740] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:40.915772 5725 consensus_queue.cc:260] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:40.916555 5725 raft_consensus.cc:3060] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:40.919137 5800 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 4.
I20260620 19:57:40.920224 5837 leader_election.cc:304] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:40.920933 5725 raft_consensus.cc:2468] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 4.
I20260620 19:57:40.921234 6067 raft_consensus.cc:2804] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:57:40.922828 6067 raft_consensus.cc:697] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 4 LEADER]: Becoming Leader. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:40.923631 6067 consensus_queue.cc:237] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:40.929525 5623 catalog_manager.cc:5696] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed reported cstate change: term changed from 3 to 4, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to cb9a18ec75324292b1c71c978353bfed (127.4.244.131). New cstate: current_term: 4 leader_uuid: "cb9a18ec75324292b1c71c978353bfed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:40.959360 6066 raft_consensus.cc:993] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:40.959749 6033 raft_consensus.cc:1081] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Signalling peer cb9a18ec75324292b1c71c978353bfed to start an election
I20260620 19:57:40.961107 5874 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69"
dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
from {username='slave'} at 127.0.0.1:47012
I20260620 19:57:40.961545 5874 raft_consensus.cc:493] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:40.961826 5874 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:40.965780 5874 raft_consensus.cc:515] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:40.967098 5874 leader_election.cc:290] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
I20260620 19:57:40.967927 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:40.968055 5800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e87edcc1a1a44c987e7077250420b69" candidate_uuid: "cb9a18ec75324292b1c71c978353bfed" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
I20260620 19:57:40.968412 5725 raft_consensus.cc:3055] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:57:40.968529 5800 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:40.968689 5725 raft_consensus.cc:740] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:40.969401 5725 consensus_queue.cc:260] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:40.970286 5725 raft_consensus.cc:3060] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:40.972723 5800 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 4.
I20260620 19:57:40.973635 5837 leader_election.cc:304] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:40.974262 6067 raft_consensus.cc:2804] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:57:40.974223 5725 raft_consensus.cc:2468] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9a18ec75324292b1c71c978353bfed in term 4.
I20260620 19:57:40.974867 6067 raft_consensus.cc:697] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 4 LEADER]: Becoming Leader. State: Replica: cb9a18ec75324292b1c71c978353bfed, State: Running, Role: LEADER
I20260620 19:57:40.975726 6067 consensus_queue.cc:237] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } }
I20260620 19:57:40.983289 5623 catalog_manager.cc:5696] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed reported cstate change: term changed from 3 to 4, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to cb9a18ec75324292b1c71c978353bfed (127.4.244.131). New cstate: current_term: 4 leader_uuid: "cb9a18ec75324292b1c71c978353bfed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:41.030033 6066 raft_consensus.cc:993] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98: : Instructing follower 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:41.030447 6033 raft_consensus.cc:1081] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Signalling peer 994e5d10524b4801a8457e3a58f2527d to start an election
I20260620 19:57:41.031810 5800 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7123643714146939d0d82713a70121e"
dest_uuid: "994e5d10524b4801a8457e3a58f2527d"
from {username='slave'} at 127.0.0.1:49402
I20260620 19:57:41.032250 5800 raft_consensus.cc:493] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:57:41.032497 5800 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:41.036773 5800 raft_consensus.cc:515] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:41.038118 5800 leader_election.cc:290] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 4 election: Requested vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), cb9a18ec75324292b1c71c978353bfed (127.4.244.131:34813)
I20260620 19:57:41.038908 5725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98"
I20260620 19:57:41.039033 5874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7123643714146939d0d82713a70121e" candidate_uuid: "994e5d10524b4801a8457e3a58f2527d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cb9a18ec75324292b1c71c978353bfed"
I20260620 19:57:41.039501 5725 raft_consensus.cc:3055] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:57:41.039546 5874 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:41.039846 5725 raft_consensus.cc:740] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dcc4c58fa6cb49a9a74281bb8e748e98, State: Running, Role: LEADER
I20260620 19:57:41.040467 5725 consensus_queue.cc:260] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:41.041298 5725 raft_consensus.cc:3060] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:57:41.043889 5874 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 4.
I20260620 19:57:41.044945 5766 leader_election.cc:304] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 994e5d10524b4801a8457e3a58f2527d, cb9a18ec75324292b1c71c978353bfed; no voters:
I20260620 19:57:41.045501 5725 raft_consensus.cc:2468] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994e5d10524b4801a8457e3a58f2527d in term 4.
I20260620 19:57:41.045584 6045 raft_consensus.cc:2804] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:57:41.046797 6045 raft_consensus.cc:697] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 4 LEADER]: Becoming Leader. State: Replica: 994e5d10524b4801a8457e3a58f2527d, State: Running, Role: LEADER
I20260620 19:57:41.047508 6045 consensus_queue.cc:237] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:41.054431 5623 catalog_manager.cc:5696] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d reported cstate change: term changed from 3 to 4, leader changed from dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129) to 994e5d10524b4801a8457e3a58f2527d (127.4.244.130). New cstate: current_term: 4 leader_uuid: "994e5d10524b4801a8457e3a58f2527d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:41.219136 5874 raft_consensus.cc:1275] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:41.220317 6045 consensus_queue.cc:1048] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:41.229702 5725 raft_consensus.cc:1275] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:57:41.231210 6045 consensus_queue.cc:1048] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:57:41.407378 5800 raft_consensus.cc:1275] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:41.408509 6074 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:41.409929 5725 raft_consensus.cc:1275] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:41.411230 6074 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:41.417585 5724 raft_consensus.cc:1275] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:41.419135 6067 consensus_queue.cc:1048] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:41.432142 5800 raft_consensus.cc:1275] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Refusing update from remote peer cb9a18ec75324292b1c71c978353bfed: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:41.433816 6067 consensus_queue.cc:1048] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [LEADER]: Connected to new peer: Peer: permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:41.588830 5725 raft_consensus.cc:1275] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:41.590271 6045 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:57:41.600834 5874 raft_consensus.cc:1275] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Refusing update from remote peer 994e5d10524b4801a8457e3a58f2527d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:57:41.602319 6045 consensus_queue.cc:1048] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:57:41.743603 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:41.749962 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:41.750387 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:42.751353 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:42.756534 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:42.756948 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:43.757956 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:43.763438 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:43.763761 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:44.764552 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:44.769763 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:44.770135 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:45.770964 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:45.777390 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:45.777921 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:46.778890 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:46.784106 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:46.784485 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:47.785331 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:47.790685 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:47.791065 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:48.792001 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:48.797111 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:48.797541 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:57:49.888211 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:57:49.893291 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260620 19:57:49.893705 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260620 19:57:50.295979 5821 debug-util.cc:398] Leaking SignalData structure 0x7b08001017a0 after lost signal to thread 5077
W20260620 19:57:50.296767 5821 debug-util.cc:398] Leaking SignalData structure 0x7b0800101e80 after lost signal to thread 5657
W20260620 19:57:50.297541 5821 debug-util.cc:398] Leaking SignalData structure 0x7b08001be0c0 after lost signal to thread 5749
W20260620 19:57:50.298429 5821 debug-util.cc:398] Leaking SignalData structure 0x7b0800015c80 after lost signal to thread 5824
W20260620 19:57:50.299539 5821 debug-util.cc:398] Leaking SignalData structure 0x7b0800097080 after lost signal to thread 5898
I20260620 19:57:50.927059 5074 auto_leader_rebalancer-test.cc:410] The leader distribution is
I20260620 19:57:50.927832 5074 tablet_server.cc:179] TabletServer@127.4.244.129:0 shutting down...
I20260620 19:57:50.961189 5074 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260620 19:57:50.962087 5074 tablet_replica.cc:333] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.963059 5074 raft_consensus.cc:2243] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Raft consensus shutting down.
I20260620 19:57:50.964197 5074 raft_consensus.cc:2272] T f1814c9aa9f640b7bc73abee51e147c3 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.967209 5074 tablet_replica.cc:333] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.967952 5074 raft_consensus.cc:2243] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:50.968485 5074 raft_consensus.cc:2272] T e06c815cc3e64a9fb26c529634a70ccb P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.971268 5074 tablet_replica.cc:333] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.972030 5074 raft_consensus.cc:2243] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:50.972586 5074 raft_consensus.cc:2272] T b7123643714146939d0d82713a70121e P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.975077 5074 tablet_replica.cc:333] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.975760 5074 raft_consensus.cc:2243] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:57:50.976694 5074 raft_consensus.cc:2272] T f90568e63c88430ea023f53fb89ac8d4 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.979298 5074 tablet_replica.cc:333] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.980026 5074 raft_consensus.cc:2243] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:50.980577 5074 raft_consensus.cc:2272] T f9112ebb71314faba51ce60093f7694f P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.982925 5074 tablet_replica.cc:333] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.983700 5074 raft_consensus.cc:2243] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:50.984267 5074 raft_consensus.cc:2272] T 16dfb3ea9ba94e66a939bfea5adb58ff P dcc4c58fa6cb49a9a74281bb8e748e98 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.986743 5074 tablet_replica.cc:333] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.987480 5074 raft_consensus.cc:2243] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 LEADER]: Raft consensus shutting down.
I20260620 19:57:50.988543 5074 raft_consensus.cc:2272] T 5271c718decc4e10895728893bf42fcf P dcc4c58fa6cb49a9a74281bb8e748e98 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.990981 5074 tablet_replica.cc:333] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.991755 5074 raft_consensus.cc:2243] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:50.992301 5074 raft_consensus.cc:2272] T fa96289638f14c039a9f769c43a5d912 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.994753 5074 tablet_replica.cc:333] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.995362 5074 raft_consensus.cc:2243] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:57:50.996454 5074 raft_consensus.cc:2272] T 965976254d4a461ba6ad6af4aaa5c441 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:50.998705 5074 tablet_replica.cc:333] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98: stopping tablet replica
I20260620 19:57:50.999325 5074 raft_consensus.cc:2243] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:50.999892 5074 raft_consensus.cc:2272] T 8e87edcc1a1a44c987e7077250420b69 P dcc4c58fa6cb49a9a74281bb8e748e98 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.037777 5074 tablet_server.cc:196] TabletServer@127.4.244.129:0 shutdown complete.
I20260620 19:57:51.060073 5074 tablet_server.cc:179] TabletServer@127.4.244.130:0 shutting down...
W20260620 19:57:51.090211 5764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260620 19:57:51.095355 5764 consensus_peers.cc:597] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d -> Peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Couldn't send request to peer dcc4c58fa6cb49a9a74281bb8e748e98. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:51.095592 5074 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260620 19:57:51.096290 5074 tablet_replica.cc:333] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.097002 5074 raft_consensus.cc:2243] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:51.097527 5074 raft_consensus.cc:2272] T e06c815cc3e64a9fb26c529634a70ccb P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.099536 5074 tablet_replica.cc:333] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.100065 5074 raft_consensus.cc:2243] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 4 LEADER]: Raft consensus shutting down.
I20260620 19:57:51.100775 5074 raft_consensus.cc:2272] T b7123643714146939d0d82713a70121e P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.102944 5074 tablet_replica.cc:333] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.103440 5074 raft_consensus.cc:2243] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:51.103870 5074 raft_consensus.cc:2272] T f1814c9aa9f640b7bc73abee51e147c3 P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.105607 5074 tablet_replica.cc:333] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.106141 5074 raft_consensus.cc:2243] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Raft consensus shutting down.
I20260620 19:57:51.106913 5074 raft_consensus.cc:2272] T f9112ebb71314faba51ce60093f7694f P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.109042 5074 tablet_replica.cc:333] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.109586 5074 raft_consensus.cc:2243] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:51.110020 5074 raft_consensus.cc:2272] T 5271c718decc4e10895728893bf42fcf P 994e5d10524b4801a8457e3a58f2527d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.112186 5074 tablet_replica.cc:333] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.112761 5074 raft_consensus.cc:2243] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 2 LEADER]: Raft consensus shutting down.
I20260620 19:57:51.113713 5074 raft_consensus.cc:2272] T 16dfb3ea9ba94e66a939bfea5adb58ff P 994e5d10524b4801a8457e3a58f2527d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260620 19:57:51.115751 5838 consensus_peers.cc:597] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed -> Peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Couldn't send request to peer dcc4c58fa6cb49a9a74281bb8e748e98. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:51.116017 5074 tablet_replica.cc:333] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.116585 5074 raft_consensus.cc:2243] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:51.116967 5074 raft_consensus.cc:2272] T fa96289638f14c039a9f769c43a5d912 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.118790 5074 tablet_replica.cc:333] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.119470 5074 raft_consensus.cc:2243] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:51.119958 5074 raft_consensus.cc:2272] T 965976254d4a461ba6ad6af4aaa5c441 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.122265 5074 tablet_replica.cc:333] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.122861 5074 raft_consensus.cc:2243] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 19:57:51.122725 5837 consensus_peers.cc:597] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed -> Peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Couldn't send request to peer 994e5d10524b4801a8457e3a58f2527d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:51.123353 5074 raft_consensus.cc:2272] T f90568e63c88430ea023f53fb89ac8d4 P 994e5d10524b4801a8457e3a58f2527d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:51.125042 5074 tablet_replica.cc:333] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d: stopping tablet replica
I20260620 19:57:51.125500 5074 raft_consensus.cc:2243] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:51.125939 5074 raft_consensus.cc:2272] T 8e87edcc1a1a44c987e7077250420b69 P 994e5d10524b4801a8457e3a58f2527d [term 4 FOLLOWER]: Raft consensus is shut down!
W20260620 19:57:51.211189 5838 consensus_peers.cc:597] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed -> Peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Couldn't send request to peer dcc4c58fa6cb49a9a74281bb8e748e98. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 19:57:51.239809 5838 consensus_peers.cc:597] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed -> Peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Couldn't send request to peer dcc4c58fa6cb49a9a74281bb8e748e98. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 19:57:51.283954 5837 consensus_peers.cc:597] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed -> Peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Couldn't send request to peer 994e5d10524b4801a8457e3a58f2527d. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 19:57:51.303839 5837 consensus_peers.cc:597] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed -> Peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Couldn't send request to peer 994e5d10524b4801a8457e3a58f2527d. Status: Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260620 19:57:52.153008 5074 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 5821)
I20260620 19:57:52.163098 6084 raft_consensus.cc:493] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dcc4c58fa6cb49a9a74281bb8e748e98)
I20260620 19:57:52.163985 6084 raft_consensus.cc:515] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:52.166559 6084 leader_election.cc:290] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
W20260620 19:57:52.170727 5837 leader_election.cc:336] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111)
W20260620 19:57:52.171417 5838 leader_election.cc:336] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111)
I20260620 19:57:52.171738 5838 leader_election.cc:304] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:52.172386 6084 raft_consensus.cc:2749] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:57:52.244820 6084 raft_consensus.cc:493] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 994e5d10524b4801a8457e3a58f2527d)
I20260620 19:57:52.245314 6084 raft_consensus.cc:515] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:52.247017 6084 leader_election.cc:290] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
W20260620 19:57:52.250487 5838 leader_election.cc:336] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111)
W20260620 19:57:52.251230 5837 leader_election.cc:336] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111)
I20260620 19:57:52.251673 5837 leader_election.cc:304] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:52.252439 6084 raft_consensus.cc:2749] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:57:52.304204 6084 raft_consensus.cc:493] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dcc4c58fa6cb49a9a74281bb8e748e98)
I20260620 19:57:52.304683 6084 raft_consensus.cc:515] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:52.306285 6084 leader_election.cc:290] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
W20260620 19:57:52.309917 5838 leader_election.cc:336] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111)
W20260620 19:57:52.310505 5837 leader_election.cc:336] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111)
I20260620 19:57:52.310880 5837 leader_election.cc:304] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:52.311446 6084 raft_consensus.cc:2749] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:57:52.352835 6084 raft_consensus.cc:493] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dcc4c58fa6cb49a9a74281bb8e748e98)
I20260620 19:57:52.353575 6084 raft_consensus.cc:515] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:52.355800 6084 leader_election.cc:290] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
W20260620 19:57:52.359293 5838 leader_election.cc:336] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111)
W20260620 19:57:52.359858 5837 leader_election.cc:336] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111)
I20260620 19:57:52.360170 5837 leader_election.cc:304] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:52.360703 6084 raft_consensus.cc:2749] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:57:52.405050 6084 raft_consensus.cc:493] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dcc4c58fa6cb49a9a74281bb8e748e98)
I20260620 19:57:52.405684 6084 raft_consensus.cc:515] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:52.407982 6084 leader_election.cc:290] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
W20260620 19:57:52.411586 5838 leader_election.cc:336] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111)
W20260620 19:57:52.412137 5837 leader_election.cc:336] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111)
I20260620 19:57:52.412431 5837 leader_election.cc:304] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:52.412972 6084 raft_consensus.cc:2749] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:57:52.472884 6084 raft_consensus.cc:493] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 994e5d10524b4801a8457e3a58f2527d)
I20260620 19:57:52.473563 6084 raft_consensus.cc:515] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:52.475765 6084 leader_election.cc:290] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
W20260620 19:57:52.479910 5837 leader_election.cc:336] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111)
W20260620 19:57:52.480358 5838 leader_election.cc:336] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111)
I20260620 19:57:52.480839 5838 leader_election.cc:304] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:52.481617 6084 raft_consensus.cc:2749] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:57:52.504347 6084 raft_consensus.cc:493] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 994e5d10524b4801a8457e3a58f2527d)
I20260620 19:57:52.504901 6084 raft_consensus.cc:515] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc4c58fa6cb49a9a74281bb8e748e98" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 34991 } } peers { permanent_uuid: "994e5d10524b4801a8457e3a58f2527d" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 33843 } } peers { permanent_uuid: "cb9a18ec75324292b1c71c978353bfed" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 34813 } }
I20260620 19:57:52.506397 6084 leader_election.cc:290] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991), 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843)
W20260620 19:57:52.510351 5838 leader_election.cc:336] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dcc4c58fa6cb49a9a74281bb8e748e98 (127.4.244.129:34991): Network error: Client connection negotiation failed: client connection to 127.4.244.129:34991: connect: Connection refused (error 111)
W20260620 19:57:52.510879 5837 leader_election.cc:336] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 994e5d10524b4801a8457e3a58f2527d (127.4.244.130:33843): Network error: Client connection negotiation failed: client connection to 127.4.244.130:33843: connect: Connection refused (error 111)
I20260620 19:57:52.511291 5837 leader_election.cc:304] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb9a18ec75324292b1c71c978353bfed; no voters: 994e5d10524b4801a8457e3a58f2527d, dcc4c58fa6cb49a9a74281bb8e748e98
I20260620 19:57:52.511969 6084 raft_consensus.cc:2749] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:57:52.603525 5074 tablet_server.cc:196] TabletServer@127.4.244.130:0 shutdown complete.
I20260620 19:57:52.627314 5074 tablet_server.cc:179] TabletServer@127.4.244.131:0 shutting down...
I20260620 19:57:52.651211 5074 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260620 19:57:52.651832 5074 tablet_replica.cc:333] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.652413 5074 raft_consensus.cc:2243] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:52.652818 5074 raft_consensus.cc:2272] T 965976254d4a461ba6ad6af4aaa5c441 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.654803 5074 tablet_replica.cc:333] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.655249 5074 raft_consensus.cc:2243] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:57:52.655970 5074 raft_consensus.cc:2272] T fa96289638f14c039a9f769c43a5d912 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.657532 5074 tablet_replica.cc:333] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.657971 5074 raft_consensus.cc:2243] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 4 LEADER]: Raft consensus shutting down.
I20260620 19:57:52.658726 5074 raft_consensus.cc:2272] T e06c815cc3e64a9fb26c529634a70ccb P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.660315 5074 tablet_replica.cc:333] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.660738 5074 raft_consensus.cc:2243] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:52.661087 5074 raft_consensus.cc:2272] T f1814c9aa9f640b7bc73abee51e147c3 P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.662585 5074 tablet_replica.cc:333] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.663049 5074 raft_consensus.cc:2243] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:52.663440 5074 raft_consensus.cc:2272] T b7123643714146939d0d82713a70121e P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.664985 5074 tablet_replica.cc:333] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.665410 5074 raft_consensus.cc:2243] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:52.665894 5074 raft_consensus.cc:2272] T f9112ebb71314faba51ce60093f7694f P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.667547 5074 tablet_replica.cc:333] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.667990 5074 raft_consensus.cc:2243] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:52.668411 5074 raft_consensus.cc:2272] T 5271c718decc4e10895728893bf42fcf P cb9a18ec75324292b1c71c978353bfed [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.670152 5074 tablet_replica.cc:333] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.670584 5074 raft_consensus.cc:2243] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:52.671028 5074 raft_consensus.cc:2272] T 16dfb3ea9ba94e66a939bfea5adb58ff P cb9a18ec75324292b1c71c978353bfed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.672489 5074 tablet_replica.cc:333] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.672931 5074 raft_consensus.cc:2243] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 4 LEADER]: Raft consensus shutting down.
I20260620 19:57:52.673635 5074 raft_consensus.cc:2272] T 8e87edcc1a1a44c987e7077250420b69 P cb9a18ec75324292b1c71c978353bfed [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.675369 5074 tablet_replica.cc:333] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed: stopping tablet replica
I20260620 19:57:52.675783 5074 raft_consensus.cc:2243] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:57:52.676162 5074 raft_consensus.cc:2272] T f90568e63c88430ea023f53fb89ac8d4 P cb9a18ec75324292b1c71c978353bfed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.704540 5074 tablet_server.cc:196] TabletServer@127.4.244.131:0 shutdown complete.
I20260620 19:57:52.723801 5074 master.cc:562] Master@127.4.244.190:33981 shutting down...
I20260620 19:57:52.746299 5074 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:57:52.746897 5074 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:57:52.747393 5074 tablet_replica.cc:333] T 00000000000000000000000000000000 P 16b6b364cb1249c2a1b9068751bcde4e: stopping tablet replica
I20260620 19:57:52.765749 5074 master.cc:584] Master@127.4.244.190:33981 shutdown complete.
994e5d10524b4801a8457e3a58f2527d 4
cb9a18ec75324292b1c71c978353bfed 1
dcc4c58fa6cb49a9a74281bb8e748e98 5
994e5d10524b4801a8457e3a58f2527d 4
cb9a18ec75324292b1c71c978353bfed 3
dcc4c58fa6cb49a9a74281bb8e748e98 3
994e5d10524b4801a8457e3a58f2527d 1
cb9a18ec75324292b1c71c978353bfed 1
dcc4c58fa6cb49a9a74281bb8e748e98 8
994e5d10524b4801a8457e3a58f2527d 3
cb9a18ec75324292b1c71c978353bfed 3
dcc4c58fa6cb49a9a74281bb8e748e98 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30358 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260620 19:57:52.807530 5074 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.244.190:33963
I20260620 19:57:52.808594 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:52.814487 6105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:52.814493 6106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:52.814846 6108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:52.816027 5074 server_base.cc:1061] running on GCE node
I20260620 19:57:52.816855 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:52.817065 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:52.817243 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985472817212 us; error 0 us; skew 500 ppm
I20260620 19:57:52.817734 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:52.820015 5074 webserver.cc:533] Webserver started at http://127.4.244.190:34131/ using document root <none> and password file <none>
I20260620 19:57:52.820494 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:52.820694 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:52.820960 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:52.822135 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/master-0-root/instance:
uuid: "8bc9d9c8cc8a4f5d911f3ec139791862"
format_stamp: "Formatted at 2026-06-20 19:57:52 on dist-test-slave-zxgc"
I20260620 19:57:52.827412 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 19:57:52.830523 6113 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:52.831254 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:57:52.831537 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/master-0-root
uuid: "8bc9d9c8cc8a4f5d911f3ec139791862"
format_stamp: "Formatted at 2026-06-20 19:57:52 on dist-test-slave-zxgc"
I20260620 19:57:52.831818 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:52.843261 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:52.844477 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:52.881155 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.190:33963
I20260620 19:57:52.881280 6164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.190:33963 every 8 connection(s)
I20260620 19:57:52.885039 6165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:52.895100 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Bootstrap starting.
I20260620 19:57:52.899467 6165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:52.903332 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: No bootstrap required, opened a new log
I20260620 19:57:52.905299 6165 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" member_type: VOTER }
I20260620 19:57:52.905694 6165 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:52.905874 6165 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc9d9c8cc8a4f5d911f3ec139791862, State: Initialized, Role: FOLLOWER
I20260620 19:57:52.906402 6165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" member_type: VOTER }
I20260620 19:57:52.906867 6165 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:57:52.907052 6165 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:57:52.907253 6165 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:52.911649 6165 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" member_type: VOTER }
I20260620 19:57:52.912092 6165 leader_election.cc:304] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bc9d9c8cc8a4f5d911f3ec139791862; no voters:
I20260620 19:57:52.913075 6165 leader_election.cc:290] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:57:52.913446 6168 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:52.914736 6168 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc9d9c8cc8a4f5d911f3ec139791862, State: Running, Role: LEADER
I20260620 19:57:52.915335 6168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" member_type: VOTER }
I20260620 19:57:52.915973 6165 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:57:52.920702 6169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" member_type: VOTER } }
I20260620 19:57:52.921398 6169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [sys.catalog]: This master's current role is: LEADER
I20260620 19:57:52.921253 6170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc9d9c8cc8a4f5d911f3ec139791862. Latest consensus state: current_term: 1 leader_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc9d9c8cc8a4f5d911f3ec139791862" member_type: VOTER } }
I20260620 19:57:52.922050 6170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862 [sys.catalog]: This master's current role is: LEADER
I20260620 19:57:52.924917 6176 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260620 19:57:52.932703 6176 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260620 19:57:52.933820 5074 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260620 19:57:52.941162 6176 catalog_manager.cc:1382] Generated new cluster ID: c794d4b26db24e38ae24ece969f87173
I20260620 19:57:52.941493 6176 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260620 19:57:52.957897 6176 catalog_manager.cc:1405] Generated new certificate authority record
I20260620 19:57:52.959122 6176 catalog_manager.cc:1539] Loading token signing keys...
I20260620 19:57:52.972855 6176 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Generated new TSK 0
I20260620 19:57:52.973433 6176 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260620 19:57:53.000453 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:53.006116 6186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:53.007551 6187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:53.009654 5074 server_base.cc:1061] running on GCE node
W20260620 19:57:53.009747 6189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:53.010833 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:53.011037 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:53.011183 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985473011169 us; error 0 us; skew 500 ppm
I20260620 19:57:53.011739 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:53.013963 5074 webserver.cc:533] Webserver started at http://127.4.244.129:34033/ using document root <none> and password file <none>
I20260620 19:57:53.014444 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:53.014676 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:53.014937 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:53.015964 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/instance:
uuid: "eb2f228ebe40440d877be6302bba2cd6"
format_stamp: "Formatted at 2026-06-20 19:57:53 on dist-test-slave-zxgc"
I20260620 19:57:53.020306 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 19:57:53.023308 6194 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.023945 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 19:57:53.024240 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root
uuid: "eb2f228ebe40440d877be6302bba2cd6"
format_stamp: "Formatted at 2026-06-20 19:57:53 on dist-test-slave-zxgc"
I20260620 19:57:53.024497 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:53.033529 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:53.034780 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:53.036120 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:57:53.038240 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:57:53.038455 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.038712 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:57:53.038902 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.093634 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.129:36725
I20260620 19:57:53.093663 6256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.129:36725 every 8 connection(s)
I20260620 19:57:53.100441 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:53.107805 6261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:53.108407 6262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:53.111042 6264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:53.112950 5074 server_base.cc:1061] running on GCE node
I20260620 19:57:53.114482 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:53.114784 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:53.114982 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985473114961 us; error 0 us; skew 500 ppm
I20260620 19:57:53.115607 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:53.115571 6257 heartbeater.cc:344] Connected to a master server at 127.4.244.190:33963
I20260620 19:57:53.116060 6257 heartbeater.cc:461] Registering TS with master...
I20260620 19:57:53.117059 6257 heartbeater.cc:507] Master 127.4.244.190:33963 requested a full tablet report, sending...
I20260620 19:57:53.118842 5074 webserver.cc:533] Webserver started at http://127.4.244.130:42369/ using document root <none> and password file <none>
I20260620 19:57:53.119463 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:53.119714 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:53.119808 6130 ts_manager.cc:194] Registered new tserver with Master: eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:57:53.120064 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:53.121598 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/instance:
uuid: "39662d01be7a44f682d160227f457661"
format_stamp: "Formatted at 2026-06-20 19:57:53 on dist-test-slave-zxgc"
I20260620 19:57:53.122179 6130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47072
I20260620 19:57:53.126819 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 19:57:53.130517 6269 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.131601 5074 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:57:53.131948 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root
uuid: "39662d01be7a44f682d160227f457661"
format_stamp: "Formatted at 2026-06-20 19:57:53 on dist-test-slave-zxgc"
I20260620 19:57:53.132254 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:53.147755 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:53.148955 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:53.150405 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:57:53.152626 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:57:53.152849 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.153088 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:57:53.153262 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.193823 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.130:42141
I20260620 19:57:53.193920 6331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.130:42141 every 8 connection(s)
I20260620 19:57:53.198470 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:57:53.205173 6335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:53.206929 6336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:57:53.209901 6338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:57:53.210682 6332 heartbeater.cc:344] Connected to a master server at 127.4.244.190:33963
I20260620 19:57:53.210922 5074 server_base.cc:1061] running on GCE node
I20260620 19:57:53.211095 6332 heartbeater.cc:461] Registering TS with master...
I20260620 19:57:53.211650 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:57:53.211831 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:57:53.211791 6332 heartbeater.cc:507] Master 127.4.244.190:33963 requested a full tablet report, sending...
I20260620 19:57:53.212088 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985473212076 us; error 0 us; skew 500 ppm
I20260620 19:57:53.212796 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:57:53.214223 6130 ts_manager.cc:194] Registered new tserver with Master: 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:53.215574 5074 webserver.cc:533] Webserver started at http://127.4.244.131:39825/ using document root <none> and password file <none>
I20260620 19:57:53.216063 6130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47082
I20260620 19:57:53.216212 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:57:53.216459 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:57:53.216763 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:57:53.218238 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/instance:
uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
format_stamp: "Formatted at 2026-06-20 19:57:53 on dist-test-slave-zxgc"
I20260620 19:57:53.222666 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:57:53.225955 6343 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.226795 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:57:53.227059 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root
uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
format_stamp: "Formatted at 2026-06-20 19:57:53 on dist-test-slave-zxgc"
I20260620 19:57:53.227358 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:57:53.246805 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:57:53.248073 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:57:53.249457 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:57:53.251659 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:57:53.251827 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.252122 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:57:53.252279 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:57:53.290362 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.131:41701
I20260620 19:57:53.290485 6405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.131:41701 every 8 connection(s)
I20260620 19:57:53.304399 6406 heartbeater.cc:344] Connected to a master server at 127.4.244.190:33963
I20260620 19:57:53.304787 6406 heartbeater.cc:461] Registering TS with master...
I20260620 19:57:53.305444 6406 heartbeater.cc:507] Master 127.4.244.190:33963 requested a full tablet report, sending...
I20260620 19:57:53.307509 6130 ts_manager.cc:194] Registered new tserver with Master: 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:53.308281 5074 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014577564s
I20260620 19:57:53.308812 6130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47092
I20260620 19:57:53.334343 6130 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47098:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:57:53.340162 6130 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:57:53.535418 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 42a982b7017a41e1b0ca419a82cacf59 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260620 19:57:53.536966 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a982b7017a41e1b0ca419a82cacf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.537840 6370 tablet_service.cc:1511] Processing CreateTablet for tablet dc6cbf763dbc4c3888feddf378ef4405 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260620 19:57:53.539220 6370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6cbf763dbc4c3888feddf378ef4405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.543746 6367 tablet_service.cc:1511] Processing CreateTablet for tablet f202638d921b4ef2b75223ca54d70e04 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260620 19:57:53.545135 6367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f202638d921b4ef2b75223ca54d70e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.547860 6369 tablet_service.cc:1511] Processing CreateTablet for tablet 1cb7a18e80e145f6913b635487750934 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260620 19:57:53.548116 6368 tablet_service.cc:1511] Processing CreateTablet for tablet 0153cc2373244ab0b2574d1de595c4a4 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260620 19:57:53.549145 6369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb7a18e80e145f6913b635487750934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.554946 6368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0153cc2373244ab0b2574d1de595c4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.560930 6365 tablet_service.cc:1511] Processing CreateTablet for tablet a00974f881fb476fb0715a861ffcf358 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260620 19:57:53.562350 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 48323077f2994fe383c12116b6e89657 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260620 19:57:53.563570 6364 tablet_service.cc:1511] Processing CreateTablet for tablet 6cda425863f04d209a457c07ec3064d1 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260620 19:57:53.564695 6222 tablet_service.cc:1511] Processing CreateTablet for tablet 42a982b7017a41e1b0ca419a82cacf59 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260620 19:57:53.566077 6222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a982b7017a41e1b0ca419a82cacf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.565874 6363 tablet_service.cc:1511] Processing CreateTablet for tablet a5739ffa3b914095862145adf0c084fb (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260620 19:57:53.567770 6218 tablet_service.cc:1511] Processing CreateTablet for tablet f202638d921b4ef2b75223ca54d70e04 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260620 19:57:53.567837 6221 tablet_service.cc:1511] Processing CreateTablet for tablet dc6cbf763dbc4c3888feddf378ef4405 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260620 19:57:53.569199 6218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f202638d921b4ef2b75223ca54d70e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.568869 6296 tablet_service.cc:1511] Processing CreateTablet for tablet dc6cbf763dbc4c3888feddf378ef4405 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260620 19:57:53.569789 6297 tablet_service.cc:1511] Processing CreateTablet for tablet 42a982b7017a41e1b0ca419a82cacf59 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260620 19:57:53.571521 6216 tablet_service.cc:1511] Processing CreateTablet for tablet a00974f881fb476fb0715a861ffcf358 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260620 19:57:53.571789 6215 tablet_service.cc:1511] Processing CreateTablet for tablet 6cda425863f04d209a457c07ec3064d1 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260620 19:57:53.571964 6295 tablet_service.cc:1511] Processing CreateTablet for tablet 1cb7a18e80e145f6913b635487750934 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260620 19:57:53.573441 6217 tablet_service.cc:1511] Processing CreateTablet for tablet 48323077f2994fe383c12116b6e89657 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260620 19:57:53.575415 6219 tablet_service.cc:1511] Processing CreateTablet for tablet 0153cc2373244ab0b2574d1de595c4a4 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260620 19:57:53.575736 6293 tablet_service.cc:1511] Processing CreateTablet for tablet f202638d921b4ef2b75223ca54d70e04 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260620 19:57:53.577342 6220 tablet_service.cc:1511] Processing CreateTablet for tablet 1cb7a18e80e145f6913b635487750934 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260620 19:57:53.562376 6365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a00974f881fb476fb0715a861ffcf358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.577880 6292 tablet_service.cc:1511] Processing CreateTablet for tablet 48323077f2994fe383c12116b6e89657 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260620 19:57:53.569695 6214 tablet_service.cc:1511] Processing CreateTablet for tablet a5739ffa3b914095862145adf0c084fb (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260620 19:57:53.580066 6291 tablet_service.cc:1511] Processing CreateTablet for tablet a00974f881fb476fb0715a861ffcf358 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260620 19:57:53.567981 6213 tablet_service.cc:1511] Processing CreateTablet for tablet 84b845bdbf9e462d9997ac5cf41386c3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260620 19:57:53.570150 6296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6cbf763dbc4c3888feddf378ef4405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.574158 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 0153cc2373244ab0b2574d1de595c4a4 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260620 19:57:53.583743 6289 tablet_service.cc:1511] Processing CreateTablet for tablet a5739ffa3b914095862145adf0c084fb (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260620 19:57:53.586047 6364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cda425863f04d209a457c07ec3064d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.586205 6288 tablet_service.cc:1511] Processing CreateTablet for tablet 84b845bdbf9e462d9997ac5cf41386c3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260620 19:57:53.588315 6216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a00974f881fb476fb0715a861ffcf358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.587950 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 6cda425863f04d209a457c07ec3064d1 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260620 19:57:53.588922 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48323077f2994fe383c12116b6e89657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.589445 6297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a982b7017a41e1b0ca419a82cacf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.593421 6289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5739ffa3b914095862145adf0c084fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.593712 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0153cc2373244ab0b2574d1de595c4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.594930 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48323077f2994fe383c12116b6e89657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.595826 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cda425863f04d209a457c07ec3064d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.598541 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5739ffa3b914095862145adf0c084fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.588536 6362 tablet_service.cc:1511] Processing CreateTablet for tablet 84b845bdbf9e462d9997ac5cf41386c3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260620 19:57:53.598902 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a00974f881fb476fb0715a861ffcf358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.600783 6292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48323077f2994fe383c12116b6e89657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.602779 6215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cda425863f04d209a457c07ec3064d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.603420 6214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5739ffa3b914095862145adf0c084fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.606911 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6cbf763dbc4c3888feddf378ef4405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.608834 6220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb7a18e80e145f6913b635487750934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.611001 6362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b845bdbf9e462d9997ac5cf41386c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.612739 6213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b845bdbf9e462d9997ac5cf41386c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.624857 6421 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:53.628405 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0153cc2373244ab0b2574d1de595c4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.629390 6288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b845bdbf9e462d9997ac5cf41386c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.632836 6295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb7a18e80e145f6913b635487750934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.633814 6293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f202638d921b4ef2b75223ca54d70e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.678331 6371 tablet_service.cc:1511] Processing CreateTablet for tablet af9232a7f82e4c3f911b856592bd6ba9 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260620 19:57:53.679702 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9232a7f82e4c3f911b856592bd6ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.685065 6422 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:53.687772 6364 tablet_service.cc:1511] Processing CreateTablet for tablet 64d372664b48432eafee6464408fcef5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260620 19:57:53.689105 6364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d372664b48432eafee6464408fcef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.692454 6362 tablet_service.cc:1511] Processing CreateTablet for tablet 97eee360f90e4475858b88d88f8efc5c (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260620 19:57:53.693661 6362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97eee360f90e4475858b88d88f8efc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.703720 6218 tablet_service.cc:1511] Processing CreateTablet for tablet 97eee360f90e4475858b88d88f8efc5c (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260620 19:57:53.704983 6213 tablet_service.cc:1511] Processing CreateTablet for tablet af9232a7f82e4c3f911b856592bd6ba9 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260620 19:57:53.710961 6218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97eee360f90e4475858b88d88f8efc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.712843 6215 tablet_service.cc:1511] Processing CreateTablet for tablet 64d372664b48432eafee6464408fcef5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260620 19:57:53.714068 6215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d372664b48432eafee6464408fcef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.714804 6369 tablet_service.cc:1511] Processing CreateTablet for tablet 8dba7c3eac4b4392830be760b4060cbc (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260620 19:57:53.715971 6369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dba7c3eac4b4392830be760b4060cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.721596 6295 tablet_service.cc:1511] Processing CreateTablet for tablet af9232a7f82e4c3f911b856592bd6ba9 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260620 19:57:53.728008 6220 tablet_service.cc:1511] Processing CreateTablet for tablet 107df92b2ae944819f7fbc4cb216d56f (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260620 19:57:53.729283 6213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9232a7f82e4c3f911b856592bd6ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.737965 6221 tablet_service.cc:1511] Processing CreateTablet for tablet 8dba7c3eac4b4392830be760b4060cbc (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260620 19:57:53.738782 6295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9232a7f82e4c3f911b856592bd6ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.740880 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 107df92b2ae944819f7fbc4cb216d56f (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260620 19:57:53.742116 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107df92b2ae944819f7fbc4cb216d56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.745854 6423 tablet_bootstrap.cc:492] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:53.746093 6288 tablet_service.cc:1511] Processing CreateTablet for tablet 107df92b2ae944819f7fbc4cb216d56f (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260620 19:57:53.747357 6288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107df92b2ae944819f7fbc4cb216d56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.747459 6363 tablet_service.cc:1511] Processing CreateTablet for tablet a56f959cee394e398810b9170b9d24be (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260620 19:57:53.748679 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56f959cee394e398810b9170b9d24be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.748754 6422 tablet_bootstrap.cc:654] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:53.750553 6220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 107df92b2ae944819f7fbc4cb216d56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.734161 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 64d372664b48432eafee6464408fcef5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260620 19:57:53.752439 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d372664b48432eafee6464408fcef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.755602 6222 tablet_service.cc:1511] Processing CreateTablet for tablet a56f959cee394e398810b9170b9d24be (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260620 19:57:53.760263 6421 tablet_bootstrap.cc:654] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:53.764827 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dba7c3eac4b4392830be760b4060cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.770447 6222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56f959cee394e398810b9170b9d24be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.771570 6289 tablet_service.cc:1511] Processing CreateTablet for tablet 97eee360f90e4475858b88d88f8efc5c (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260620 19:57:53.772817 6289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97eee360f90e4475858b88d88f8efc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.775552 6217 tablet_service.cc:1511] Processing CreateTablet for tablet 1c06fb2d65824956840c8faf71df4cc9 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260620 19:57:53.776724 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c06fb2d65824956840c8faf71df4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.779004 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 1c06fb2d65824956840c8faf71df4cc9 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260620 19:57:53.780335 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c06fb2d65824956840c8faf71df4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.783914 6219 tablet_service.cc:1511] Processing CreateTablet for tablet e617e76f53974a7189d97fe9f8e69a8b (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260620 19:57:53.785456 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e617e76f53974a7189d97fe9f8e69a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.793259 6293 tablet_service.cc:1511] Processing CreateTablet for tablet a56f959cee394e398810b9170b9d24be (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260620 19:57:53.794783 6293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56f959cee394e398810b9170b9d24be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.770177 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 8dba7c3eac4b4392830be760b4060cbc (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260620 19:57:53.801826 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dba7c3eac4b4392830be760b4060cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.819207 6423 tablet_bootstrap.cc:654] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:53.823933 6297 tablet_service.cc:1511] Processing CreateTablet for tablet 1c06fb2d65824956840c8faf71df4cc9 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
W20260620 19:57:53.826849 6117 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 46115.2us
W20260620 19:57:53.828049 6115 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 45465.1us
I20260620 19:57:53.852974 6297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c06fb2d65824956840c8faf71df4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.877794 6371 tablet_service.cc:1511] Processing CreateTablet for tablet e617e76f53974a7189d97fe9f8e69a8b (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260620 19:57:53.879055 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e617e76f53974a7189d97fe9f8e69a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.876767 6290 tablet_service.cc:1511] Processing CreateTablet for tablet e617e76f53974a7189d97fe9f8e69a8b (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260620 19:57:53.887727 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e617e76f53974a7189d97fe9f8e69a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.895435 6297 tablet_service.cc:1511] Processing CreateTablet for tablet 5d113f5ff7f842158ba3c35212f7bdf3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260620 19:57:53.896819 6297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d113f5ff7f842158ba3c35212f7bdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.902281 6363 tablet_service.cc:1511] Processing CreateTablet for tablet 5d113f5ff7f842158ba3c35212f7bdf3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260620 19:57:53.903563 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d113f5ff7f842158ba3c35212f7bdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.914341 6215 tablet_service.cc:1511] Processing CreateTablet for tablet ddb66400acd140a5b26f1a19f2abd18e (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260620 19:57:53.915670 6215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb66400acd140a5b26f1a19f2abd18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.917985 6421 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:53.918394 6421 ts_tablet_manager.cc:1403] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.295s user 0.048s sys 0.031s
I20260620 19:57:53.920567 6421 raft_consensus.cc:359] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:53.921116 6421 raft_consensus.cc:385] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:53.921378 6421 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:53.922053 6421 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:53.923138 6422 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:53.923823 6422 ts_tablet_manager.cc:1403] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.239s user 0.010s sys 0.010s
I20260620 19:57:53.927156 6422 raft_consensus.cc:359] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:53.928023 6422 raft_consensus.cc:385] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260620 19:57:53.928315 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:57:53.929387 6422 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:53.930408 6219 tablet_service.cc:1511] Processing CreateTablet for tablet eb61840b191946f9a272fb1b78f28cec (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260620 19:57:53.934818 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb61840b191946f9a272fb1b78f28cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.931453 6371 tablet_service.cc:1511] Processing CreateTablet for tablet ddb66400acd140a5b26f1a19f2abd18e (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260620 19:57:53.935766 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb66400acd140a5b26f1a19f2abd18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.934921 6422 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:53.911721 6221 tablet_service.cc:1511] Processing CreateTablet for tablet 5d113f5ff7f842158ba3c35212f7bdf3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260620 19:57:53.933284 6297 tablet_service.cc:1511] Processing CreateTablet for tablet ddb66400acd140a5b26f1a19f2abd18e (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260620 19:57:53.935644 6294 tablet_service.cc:1511] Processing CreateTablet for tablet eb61840b191946f9a272fb1b78f28cec (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260620 19:57:53.943742 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d113f5ff7f842158ba3c35212f7bdf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.944766 6213 tablet_service.cc:1511] Processing CreateTablet for tablet db666bffcf904fa59d91db75dc91c8b0 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260620 19:57:53.946029 6213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db666bffcf904fa59d91db75dc91c8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.944257 6297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb66400acd140a5b26f1a19f2abd18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.946904 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb61840b191946f9a272fb1b78f28cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.938975 6366 tablet_service.cc:1511] Processing CreateTablet for tablet eb61840b191946f9a272fb1b78f28cec (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260620 19:57:53.948335 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb61840b191946f9a272fb1b78f28cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.952749 6293 tablet_service.cc:1511] Processing CreateTablet for tablet db666bffcf904fa59d91db75dc91c8b0 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260620 19:57:53.953997 6293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db666bffcf904fa59d91db75dc91c8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.958894 6363 tablet_service.cc:1511] Processing CreateTablet for tablet db666bffcf904fa59d91db75dc91c8b0 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260620 19:57:53.960139 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db666bffcf904fa59d91db75dc91c8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.932435 6423 tablet_bootstrap.cc:492] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:53.970865 6423 ts_tablet_manager.cc:1403] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.225s user 0.010s sys 0.006s
I20260620 19:57:53.974108 6423 raft_consensus.cc:359] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:53.975610 6423 raft_consensus.cc:385] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:53.975894 6423 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:53.977855 6421 ts_tablet_manager.cc:1434] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.059s user 0.019s sys 0.017s
I20260620 19:57:53.978567 6406 heartbeater.cc:499] Master 127.4.244.190:33963 was elected leader, sending a full tablet report...
I20260620 19:57:53.978876 6421 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:53.979597 6422 ts_tablet_manager.cc:1434] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.055s user 0.006s sys 0.007s
I20260620 19:57:53.979458 6291 tablet_service.cc:1511] Processing CreateTablet for tablet aacb05a304e442a7bd73148aff5f4a66 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260620 19:57:53.980289 6257 heartbeater.cc:499] Master 127.4.244.190:33963 was elected leader, sending a full tablet report...
I20260620 19:57:53.980718 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacb05a304e442a7bd73148aff5f4a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.978572 6217 tablet_service.cc:1511] Processing CreateTablet for tablet aacb05a304e442a7bd73148aff5f4a66 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260620 19:57:53.984402 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacb05a304e442a7bd73148aff5f4a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.985230 6421 tablet_bootstrap.cc:654] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:53.983150 6368 tablet_service.cc:1511] Processing CreateTablet for tablet aacb05a304e442a7bd73148aff5f4a66 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260620 19:57:53.988258 6368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacb05a304e442a7bd73148aff5f4a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.983157 6422 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:53.996132 6220 tablet_service.cc:1511] Processing CreateTablet for tablet 5c21aa29d76d48fc933f76007685dae1 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260620 19:57:53.997370 6220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c21aa29d76d48fc933f76007685dae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:53.998101 6288 tablet_service.cc:1511] Processing CreateTablet for tablet 5c21aa29d76d48fc933f76007685dae1 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260620 19:57:53.999429 6288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c21aa29d76d48fc933f76007685dae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.004956 6427 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:54.019584 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 5c21aa29d76d48fc933f76007685dae1 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260620 19:57:54.020929 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c21aa29d76d48fc933f76007685dae1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:54.044780 6407 tablet.cc:2406] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:57:54.005460 6427 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:54.051944 6422 tablet_bootstrap.cc:654] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.005841 6423 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.065968 6429 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:54.066425 6429 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.067756 6215 tablet_service.cc:1511] Processing CreateTablet for tablet 79a07a8804784405920baab70a535338 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260620 19:57:54.069516 6215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a07a8804784405920baab70a535338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.069828 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 79a07a8804784405920baab70a535338 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260620 19:57:54.071085 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a07a8804784405920baab70a535338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.075712 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 79a07a8804784405920baab70a535338 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260620 19:57:54.077329 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a07a8804784405920baab70a535338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.087899 6422 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:54.088004 6221 tablet_service.cc:1511] Processing CreateTablet for tablet 31d307880255440b9b2c96eee7bb9f28 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260620 19:57:54.088606 6422 ts_tablet_manager.cc:1403] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.003s
I20260620 19:57:54.089427 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d307880255440b9b2c96eee7bb9f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.083928 6429 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.101729 6427 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:57:54.110766 6291 tablet_service.cc:1511] Processing CreateTablet for tablet 31d307880255440b9b2c96eee7bb9f28 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260620 19:57:54.112012 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d307880255440b9b2c96eee7bb9f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.132035 6332 heartbeater.cc:499] Master 127.4.244.190:33963 was elected leader, sending a full tablet report...
I20260620 19:57:54.138455 6421 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:54.138962 6421 ts_tablet_manager.cc:1403] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.160s user 0.022s sys 0.025s
I20260620 19:57:54.145344 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 31d307880255440b9b2c96eee7bb9f28 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260620 19:57:54.145862 6421 raft_consensus.cc:359] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.150976 6423 ts_tablet_manager.cc:1434] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.172s user 0.012s sys 0.018s
I20260620 19:57:54.155563 6423 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:54.146626 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d307880255440b9b2c96eee7bb9f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.161373 6423 tablet_bootstrap.cc:654] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.166282 6220 tablet_service.cc:1511] Processing CreateTablet for tablet 65e57f1c5f9f437db061394645beec68 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260620 19:57:54.167526 6220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e57f1c5f9f437db061394645beec68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.168210 6288 tablet_service.cc:1511] Processing CreateTablet for tablet 65e57f1c5f9f437db061394645beec68 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260620 19:57:54.169401 6288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e57f1c5f9f437db061394645beec68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.146576 6421 raft_consensus.cc:385] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.132494 6422 raft_consensus.cc:359] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.172411 6422 raft_consensus.cc:385] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.179720 6422 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.180564 6422 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.182983 6422 ts_tablet_manager.cc:1434] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.094s user 0.009s sys 0.001s
I20260620 19:57:54.182907 6368 tablet_service.cc:1511] Processing CreateTablet for tablet 65e57f1c5f9f437db061394645beec68 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260620 19:57:54.184303 6368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e57f1c5f9f437db061394645beec68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.182993 6217 tablet_service.cc:1511] Processing CreateTablet for tablet 92212144534e4394844003016ba67a47 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260620 19:57:54.185860 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 92212144534e4394844003016ba67a47 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260620 19:57:54.187060 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92212144534e4394844003016ba67a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.188558 6363 tablet_service.cc:1511] Processing CreateTablet for tablet 92212144534e4394844003016ba67a47 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260620 19:57:54.195787 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92212144534e4394844003016ba67a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.190068 6215 tablet_service.cc:1511] Processing CreateTablet for tablet 72968f173fdb4542920e97a940db9668 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260620 19:57:54.171768 6421 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.199783 6421 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.202234 6421 ts_tablet_manager.cc:1434] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.063s user 0.000s sys 0.006s
I20260620 19:57:54.208170 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 72968f173fdb4542920e97a940db9668 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260620 19:57:54.209283 6213 tablet_service.cc:1511] Processing CreateTablet for tablet eae1222c89dd47e78f503964b6a95973 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260620 19:57:54.211701 6422 tablet_bootstrap.cc:492] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:54.212877 6215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72968f173fdb4542920e97a940db9668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.213265 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72968f173fdb4542920e97a940db9668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.213479 6213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae1222c89dd47e78f503964b6a95973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.215620 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92212144534e4394844003016ba67a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.219074 6428 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:54.219527 6428 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.220268 6421 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:54.224195 6428 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.255036 6291 tablet_service.cc:1511] Processing CreateTablet for tablet 72968f173fdb4542920e97a940db9668 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260620 19:57:54.256675 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72968f173fdb4542920e97a940db9668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.259220 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:54.261204 6345 leader_election.cc:343] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.266677 6288 tablet_service.cc:1511] Processing CreateTablet for tablet eae1222c89dd47e78f503964b6a95973 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260620 19:57:54.268280 6288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae1222c89dd47e78f503964b6a95973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.270505 6421 tablet_bootstrap.cc:654] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.282148 6423 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:54.282665 6423 ts_tablet_manager.cc:1403] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.127s user 0.015s sys 0.017s
I20260620 19:57:54.285495 6423 raft_consensus.cc:359] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.286232 6423 raft_consensus.cc:385] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.286518 6423 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.287201 6423 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.289822 6423 ts_tablet_manager.cc:1434] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:57:54.294637 6422 tablet_bootstrap.cc:654] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.294970 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:54.295228 6221 tablet_service.cc:1511] Processing CreateTablet for tablet 8b0538f91fad4439a7d30adbf2f83b57 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260620 19:57:54.296522 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0538f91fad4439a7d30adbf2f83b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.296921 6366 tablet_service.cc:1511] Processing CreateTablet for tablet eae1222c89dd47e78f503964b6a95973 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260620 19:57:54.298445 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae1222c89dd47e78f503964b6a95973. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:54.308676 6198 leader_election.cc:343] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.304950 6423 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:54.316583 6423 tablet_bootstrap.cc:654] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.320223 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:54.321818 6271 leader_election.cc:343] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.323619 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 8b0538f91fad4439a7d30adbf2f83b57 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260620 19:57:54.331905 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:54.332341 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0538f91fad4439a7d30adbf2f83b57. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:54.333976 6273 leader_election.cc:343] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.334522 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:54.338982 6433 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:54.354928 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:54.355028 6363 tablet_service.cc:1511] Processing CreateTablet for tablet 8b0538f91fad4439a7d30adbf2f83b57 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260620 19:57:54.356540 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0538f91fad4439a7d30adbf2f83b57. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:54.357250 6344 leader_election.cc:343] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.357673 6344 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:54.358729 6427 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:54.359002 6291 tablet_service.cc:1511] Processing CreateTablet for tablet c5c97b15b2f74ff2a06fde66e407b188 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260620 19:57:54.358917 6217 tablet_service.cc:1511] Processing CreateTablet for tablet c5c97b15b2f74ff2a06fde66e407b188 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260620 19:57:54.360236 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c97b15b2f74ff2a06fde66e407b188. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.362193 6368 tablet_service.cc:1511] Processing CreateTablet for tablet c5c97b15b2f74ff2a06fde66e407b188 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260620 19:57:54.362705 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 7a566dbaea0746f4bc76e04b5072e045 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260620 19:57:54.363255 6220 tablet_service.cc:1511] Processing CreateTablet for tablet 7a566dbaea0746f4bc76e04b5072e045 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260620 19:57:54.363917 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a566dbaea0746f4bc76e04b5072e045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.365067 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 7a566dbaea0746f4bc76e04b5072e045 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260620 19:57:54.366130 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a566dbaea0746f4bc76e04b5072e045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.365909 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:54.360803 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c97b15b2f74ff2a06fde66e407b188. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.368008 6307 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:54.369225 6368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c97b15b2f74ff2a06fde66e407b188. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.388799 6220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a566dbaea0746f4bc76e04b5072e045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.390093 6421 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:54.390542 6421 ts_tablet_manager.cc:1403] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.170s user 0.006s sys 0.014s
I20260620 19:57:54.393653 6421 raft_consensus.cc:359] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.394320 6421 raft_consensus.cc:385] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.394624 6421 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.395138 6195 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:54.395973 6428 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:54.396318 6428 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:54.398906 6428 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.395468 6421 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.403228 6423 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:54.403713 6423 ts_tablet_manager.cc:1403] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.113s user 0.019s sys 0.022s
I20260620 19:57:54.406594 6423 raft_consensus.cc:359] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.408246 6423 raft_consensus.cc:385] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.408555 6423 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.408499 6294 tablet_service.cc:1511] Processing CreateTablet for tablet a1858fc28e4948cc8e3c29e06357d166 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260620 19:57:54.409384 6423 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.409917 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1858fc28e4948cc8e3c29e06357d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.412117 6366 tablet_service.cc:1511] Processing CreateTablet for tablet a1858fc28e4948cc8e3c29e06357d166 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260620 19:57:54.406550 6428 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.418422 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 43e3501e8f31428cb18618fc6a8e1f59 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260620 19:57:54.420392 6428 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.421310 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:54.421865 6307 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.422098 6421 ts_tablet_manager.cc:1434] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20260620 19:57:54.413558 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1858fc28e4948cc8e3c29e06357d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.414718 6423 ts_tablet_manager.cc:1434] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260620 19:57:54.423025 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:54.415987 6213 tablet_service.cc:1511] Processing CreateTablet for tablet 43e3501e8f31428cb18618fc6a8e1f59 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260620 19:57:54.424578 6213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43e3501e8f31428cb18618fc6a8e1f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.417991 6221 tablet_service.cc:1511] Processing CreateTablet for tablet a1858fc28e4948cc8e3c29e06357d166 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260620 19:57:54.423028 6422 tablet_bootstrap.cc:492] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:54.426296 6422 ts_tablet_manager.cc:1403] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.215s user 0.010s sys 0.012s
I20260620 19:57:54.426105 6215 tablet_service.cc:1511] Processing CreateTablet for tablet bb9f062c40bd4cb68d470ec9b5e63a92 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260620 19:57:54.427321 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43e3501e8f31428cb18618fc6a8e1f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.427372 6215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9f062c40bd4cb68d470ec9b5e63a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.416937 6291 tablet_service.cc:1511] Processing CreateTablet for tablet 43e3501e8f31428cb18618fc6a8e1f59 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260620 19:57:54.429271 6422 raft_consensus.cc:359] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.429977 6422 raft_consensus.cc:385] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.429924 6307 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
W20260620 19:57:54.430954 6198 leader_election.cc:343] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.430411 6422 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.433183 6422 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.434638 6290 tablet_service.cc:1511] Processing CreateTablet for tablet bb9f062c40bd4cb68d470ec9b5e63a92 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260620 19:57:54.435356 6422 ts_tablet_manager.cc:1434] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260620 19:57:54.436352 6422 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:54.440352 6219 tablet_service.cc:1511] Processing CreateTablet for tablet cfeb1fb4ea6746dbaf0509f4097e00ab (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260620 19:57:54.440415 6368 tablet_service.cc:1511] Processing CreateTablet for tablet bb9f062c40bd4cb68d470ec9b5e63a92 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260620 19:57:54.441656 6195 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:54.442822 6368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9f062c40bd4cb68d470ec9b5e63a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.444141 6369 tablet_service.cc:1511] Processing CreateTablet for tablet cfeb1fb4ea6746dbaf0509f4097e00ab (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260620 19:57:54.444437 6288 tablet_service.cc:1511] Processing CreateTablet for tablet cfeb1fb4ea6746dbaf0509f4097e00ab (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260620 19:57:54.445957 6369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfeb1fb4ea6746dbaf0509f4097e00ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.446858 6218 tablet_service.cc:1511] Processing CreateTablet for tablet 8cf0b0e2dafa4512b82ecefda3013d79 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260620 19:57:54.448321 6365 tablet_service.cc:1511] Processing CreateTablet for tablet 8cf0b0e2dafa4512b82ecefda3013d79 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260620 19:57:54.449570 6365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cf0b0e2dafa4512b82ecefda3013d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.449685 6297 tablet_service.cc:1511] Processing CreateTablet for tablet 8cf0b0e2dafa4512b82ecefda3013d79 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260620 19:57:54.450547 6433 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:54.451041 6433 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.453622 6433 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.454298 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:54.454942 6232 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:54.455957 6297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cf0b0e2dafa4512b82ecefda3013d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.456125 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:54.457358 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9f062c40bd4cb68d470ec9b5e63a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.459496 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:54.460105 6381 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:54.462785 6433 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:54.463173 6433 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:54.463507 6433 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.466830 6428 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:54.467173 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43e3501e8f31428cb18618fc6a8e1f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.453704 6421 tablet_bootstrap.cc:492] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:54.439203 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1858fc28e4948cc8e3c29e06357d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.471166 6218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cf0b0e2dafa4512b82ecefda3013d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.471168 6288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfeb1fb4ea6746dbaf0509f4097e00ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.485857 6222 tablet_service.cc:1511] Processing CreateTablet for tablet 03268245eb53439a9b88defe52240125 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260620 19:57:54.471817 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfeb1fb4ea6746dbaf0509f4097e00ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.491916 6364 tablet_service.cc:1511] Processing CreateTablet for tablet 03268245eb53439a9b88defe52240125 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260620 19:57:54.492093 6293 tablet_service.cc:1511] Processing CreateTablet for tablet 03268245eb53439a9b88defe52240125 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260620 19:57:54.493291 6293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03268245eb53439a9b88defe52240125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.493286 6364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03268245eb53439a9b88defe52240125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.471349 6423 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:54.495129 6222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03268245eb53439a9b88defe52240125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.495961 6433 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.515507 6216 tablet_service.cc:1511] Processing CreateTablet for tablet 93c3314daf2a488caf8f0d3a805f44cd (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260620 19:57:54.516800 6216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c3314daf2a488caf8f0d3a805f44cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.513934 6428 raft_consensus.cc:697] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:54.522635 6428 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.524016 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:54.524578 6232 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.529014 6433 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.530128 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:54.530753 6381 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.535403 6449 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:54.535917 6449 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.537149 6421 tablet_bootstrap.cc:654] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.543149 6232 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:54.547009 6449 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.549016 6422 tablet_bootstrap.cc:654] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.551084 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:54.552727 6307 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:54.550953 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:54.556437 6195 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:57:54.557490 6198 leader_election.cc:343] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.559888 6128 catalog_manager.cc:5696] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:54.567126 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:54.567955 6433 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:54.568411 6433 raft_consensus.cc:697] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:54.569190 6433 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.569454 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 93c3314daf2a488caf8f0d3a805f44cd (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260620 19:57:54.570719 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c3314daf2a488caf8f0d3a805f44cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.572021 6369 tablet_service.cc:1511] Processing CreateTablet for tablet 93c3314daf2a488caf8f0d3a805f44cd (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260620 19:57:54.577521 6423 tablet_bootstrap.cc:654] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.578742 6369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93c3314daf2a488caf8f0d3a805f44cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.564188 6428 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:54.556403 6381 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:54.591517 6428 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:54.595654 6428 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.596119 6427 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:54.596753 6427 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.601828 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:54.603030 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:54.604199 6427 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:57:54.605468 6345 leader_election.cc:343] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:57:54.606947 6344 leader_election.cc:343] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.607523 6344 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:54.608918 6427 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:54.628360 6422 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:54.628876 6422 ts_tablet_manager.cc:1403] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.193s user 0.014s sys 0.030s
I20260620 19:57:54.632159 6218 tablet_service.cc:1511] Processing CreateTablet for tablet 5ace7015fc514a909c3139c9e717b387 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260620 19:57:54.632445 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 5ace7015fc514a909c3139c9e717b387 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260620 19:57:54.633400 6218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ace7015fc514a909c3139c9e717b387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.633816 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ace7015fc514a909c3139c9e717b387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.636376 6422 raft_consensus.cc:359] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.637166 6422 raft_consensus.cc:385] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.637506 6422 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.645372 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 5ace7015fc514a909c3139c9e717b387 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260620 19:57:54.646643 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ace7015fc514a909c3139c9e717b387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.642009 6129 catalog_manager.cc:5696] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:54.638266 6422 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.653765 6428 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.654721 6422 ts_tablet_manager.cc:1434] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.026s user 0.006s sys 0.003s
I20260620 19:57:54.660713 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 758c2974c107466bbcc6c41d00de7b18 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260620 19:57:54.661522 6421 tablet_bootstrap.cc:492] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:54.661823 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758c2974c107466bbcc6c41d00de7b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.661983 6421 ts_tablet_manager.cc:1403] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.208s user 0.009s sys 0.019s
I20260620 19:57:54.662834 6222 tablet_service.cc:1511] Processing CreateTablet for tablet 098d4ada753442e1899d391a485095aa (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260620 19:57:54.664218 6222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098d4ada753442e1899d391a485095aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.664673 6421 raft_consensus.cc:359] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.661652 6423 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:54.665352 6421 raft_consensus.cc:385] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.665665 6421 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.665817 6423 ts_tablet_manager.cc:1403] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.195s user 0.008s sys 0.015s
I20260620 19:57:54.665592 6219 tablet_service.cc:1511] Processing CreateTablet for tablet 758c2974c107466bbcc6c41d00de7b18 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260620 19:57:54.666953 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758c2974c107466bbcc6c41d00de7b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.670856 6423 raft_consensus.cc:359] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.671613 6423 raft_consensus.cc:385] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:54.671890 6423 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:54.670845 6422 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:54.672495 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:54.673035 6307 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.680085 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 758c2974c107466bbcc6c41d00de7b18 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260620 19:57:54.681365 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758c2974c107466bbcc6c41d00de7b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.684134 6288 tablet_service.cc:1511] Processing CreateTablet for tablet 098d4ada753442e1899d391a485095aa (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260620 19:57:54.685756 6288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098d4ada753442e1899d391a485095aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.699609 6428 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.700943 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:54.672612 6423 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.707007 6222 tablet_service.cc:1511] Processing CreateTablet for tablet 8b96addaf2b149a2ba7decbb292167e5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260620 19:57:54.716265 6222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b96addaf2b149a2ba7decbb292167e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.715104 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 8b96addaf2b149a2ba7decbb292167e5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260620 19:57:54.716615 6291 tablet_service.cc:1511] Processing CreateTablet for tablet 6d08b9c2da1448139f24e1d94c9616f3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260620 19:57:54.717515 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b96addaf2b149a2ba7decbb292167e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.716615 6221 tablet_service.cc:1511] Processing CreateTablet for tablet 6d08b9c2da1448139f24e1d94c9616f3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260620 19:57:54.718703 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d08b9c2da1448139f24e1d94c9616f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.718457 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 6d08b9c2da1448139f24e1d94c9616f3 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260620 19:57:54.718950 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d08b9c2da1448139f24e1d94c9616f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.719632 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d08b9c2da1448139f24e1d94c9616f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.721776 6365 tablet_service.cc:1511] Processing CreateTablet for tablet 098d4ada753442e1899d391a485095aa (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260620 19:57:54.722958 6365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098d4ada753442e1899d391a485095aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.737346 6369 tablet_service.cc:1511] Processing CreateTablet for tablet 8b96addaf2b149a2ba7decbb292167e5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260620 19:57:54.738034 6219 tablet_service.cc:1511] Processing CreateTablet for tablet c686486d28e847d0a66123f171f1149f (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260620 19:57:54.738116 6297 tablet_service.cc:1511] Processing CreateTablet for tablet c686486d28e847d0a66123f171f1149f (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260620 19:57:54.739382 6297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c686486d28e847d0a66123f171f1149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.740007 6423 ts_tablet_manager.cc:1434] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20260620 19:57:54.740971 6423 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:54.738543 6369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b96addaf2b149a2ba7decbb292167e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.741406 6366 tablet_service.cc:1511] Processing CreateTablet for tablet c686486d28e847d0a66123f171f1149f (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260620 19:57:54.702672 6421 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.747762 6381 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:54.749192 6421 ts_tablet_manager.cc:1434] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.087s user 0.002s sys 0.006s
I20260620 19:57:54.750201 6421 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:54.741441 6307 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:54.752745 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c686486d28e847d0a66123f171f1149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.754348 6195 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:54.755182 6428 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:54.755669 6428 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:54.756462 6428 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.757500 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c686486d28e847d0a66123f171f1149f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.762152 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 8504518fd53942459d685f3abcbae2eb (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260620 19:57:54.771330 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8504518fd53942459d685f3abcbae2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.763331 6422 tablet_bootstrap.cc:654] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.764673 6381 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:54.780309 6216 tablet_service.cc:1511] Processing CreateTablet for tablet 8504518fd53942459d685f3abcbae2eb (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260620 19:57:54.781606 6216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8504518fd53942459d685f3abcbae2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.782198 6217 tablet_service.cc:1511] Processing CreateTablet for tablet 2e56dc56fbc548ac997024a1b2e2ea1a (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260620 19:57:54.788677 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 8504518fd53942459d685f3abcbae2eb (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260620 19:57:54.791651 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e56dc56fbc548ac997024a1b2e2ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.793937 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 2e56dc56fbc548ac997024a1b2e2ea1a (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260620 19:57:54.797353 6421 tablet_bootstrap.cc:654] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.808576 6218 tablet_service.cc:1511] Processing CreateTablet for tablet 35398db5947a409b98b0d62a257ece06 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260620 19:57:54.809933 6218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35398db5947a409b98b0d62a257ece06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.811326 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8504518fd53942459d685f3abcbae2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.815478 6290 tablet_service.cc:1511] Processing CreateTablet for tablet 2e56dc56fbc548ac997024a1b2e2ea1a (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260620 19:57:54.816470 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e56dc56fbc548ac997024a1b2e2ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.834841 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 35398db5947a409b98b0d62a257ece06 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260620 19:57:54.836016 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35398db5947a409b98b0d62a257ece06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.840374 6221 tablet_service.cc:1511] Processing CreateTablet for tablet 34bcb03937cb4cea98433c0a226855de (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260620 19:57:54.842824 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e56dc56fbc548ac997024a1b2e2ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.841658 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34bcb03937cb4cea98433c0a226855de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.857671 6364 tablet_service.cc:1511] Processing CreateTablet for tablet 35398db5947a409b98b0d62a257ece06 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260620 19:57:54.858960 6364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35398db5947a409b98b0d62a257ece06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.860085 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 34bcb03937cb4cea98433c0a226855de (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260620 19:57:54.861096 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34bcb03937cb4cea98433c0a226855de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.865119 6220 tablet_service.cc:1511] Processing CreateTablet for tablet 92d22c99dfc64cc1bb999795e691d1ad (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260620 19:57:54.865247 6215 tablet_service.cc:1511] Processing CreateTablet for tablet cc7845798261439d855e526604de104a (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260620 19:57:54.877542 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 92d22c99dfc64cc1bb999795e691d1ad (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260620 19:57:54.877908 6214 tablet_service.cc:1511] Processing CreateTablet for tablet a506f4812fdc4ae2b57cb125ac69acb6 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260620 19:57:54.877923 6215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7845798261439d855e526604de104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.865248 6213 tablet_service.cc:1511] Processing CreateTablet for tablet ee78433ca73b4ea8b2b24535036f6369 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260620 19:57:54.878870 6128 catalog_manager.cc:5696] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:54.879982 6368 tablet_service.cc:1511] Processing CreateTablet for tablet cc7845798261439d855e526604de104a (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260620 19:57:54.880371 6213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee78433ca73b4ea8b2b24535036f6369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.881049 6423 tablet_bootstrap.cc:654] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:54.881601 6363 tablet_service.cc:1511] Processing CreateTablet for tablet ee78433ca73b4ea8b2b24535036f6369 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260620 19:57:54.875842 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 34bcb03937cb4cea98433c0a226855de (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260620 19:57:54.882864 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34bcb03937cb4cea98433c0a226855de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.883095 6220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92d22c99dfc64cc1bb999795e691d1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.886902 6214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a506f4812fdc4ae2b57cb125ac69acb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.887437 6368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7845798261439d855e526604de104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.889210 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92d22c99dfc64cc1bb999795e691d1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.891147 6433 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:54.891619 6433 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:54.892589 6363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee78433ca73b4ea8b2b24535036f6369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.893613 6433 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:54.896721 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:54.898521 6362 tablet_service.cc:1511] Processing CreateTablet for tablet a506f4812fdc4ae2b57cb125ac69acb6 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260620 19:57:54.899793 6362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a506f4812fdc4ae2b57cb125ac69acb6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:54.904819 6271 leader_election.cc:343] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.906333 6290 tablet_service.cc:1511] Processing CreateTablet for tablet cc7845798261439d855e526604de104a (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260620 19:57:54.907433 6290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7845798261439d855e526604de104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.907881 6295 tablet_service.cc:1511] Processing CreateTablet for tablet 9d4b5cc9261e42c381b94cd3961f1cae (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260620 19:57:54.909247 6295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4b5cc9261e42c381b94cd3961f1cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.909744 6289 tablet_service.cc:1511] Processing CreateTablet for tablet 99e667a051414b87955b78b2520409b5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260620 19:57:54.910889 6289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e667a051414b87955b78b2520409b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.910759 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:54.911585 6297 tablet_service.cc:1511] Processing CreateTablet for tablet ee78433ca73b4ea8b2b24535036f6369 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260620 19:57:54.912688 6273 leader_election.cc:343] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:54.913131 6273 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:54.913465 6293 tablet_service.cc:1511] Processing CreateTablet for tablet a506f4812fdc4ae2b57cb125ac69acb6 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260620 19:57:54.914263 6433 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:54.915377 6297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee78433ca73b4ea8b2b24535036f6369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.917186 6293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a506f4812fdc4ae2b57cb125ac69acb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.919925 6294 tablet_service.cc:1511] Processing CreateTablet for tablet 92d22c99dfc64cc1bb999795e691d1ad (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260620 19:57:54.921577 6294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92d22c99dfc64cc1bb999795e691d1ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:54.953174 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:57:54.956835 6292 tablet_service.cc:1511] Processing CreateTablet for tablet 52c91ae8424d495f92c57a6fcfc235d6 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260620 19:57:54.961298 6217 tablet_service.cc:1511] Processing CreateTablet for tablet 99e667a051414b87955b78b2520409b5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260620 19:57:54.976025 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e667a051414b87955b78b2520409b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.974725 6219 tablet_service.cc:1511] Processing CreateTablet for tablet 52c91ae8424d495f92c57a6fcfc235d6 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20260620 19:57:54.979799 6459 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260620 19:57:54.985152 6422 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:54.987432 6296 tablet_service.cc:1511] Processing CreateTablet for tablet 1e39bd0f2de444d2acf683f9f18716da (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260620 19:57:54.991334 6222 tablet_service.cc:1511] Processing CreateTablet for tablet 9d4b5cc9261e42c381b94cd3961f1cae (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260620 19:57:55.008193 6370 tablet_service.cc:1511] Processing CreateTablet for tablet 1e39bd0f2de444d2acf683f9f18716da (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260620 19:57:55.009483 6370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e39bd0f2de444d2acf683f9f18716da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.011245 6366 tablet_service.cc:1511] Processing CreateTablet for tablet 52c91ae8424d495f92c57a6fcfc235d6 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260620 19:57:55.012342 6366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c91ae8424d495f92c57a6fcfc235d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.018882 6371 tablet_service.cc:1511] Processing CreateTablet for tablet 9d4b5cc9261e42c381b94cd3961f1cae (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260620 19:57:55.021180 6371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4b5cc9261e42c381b94cd3961f1cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.993795 6292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c91ae8424d495f92c57a6fcfc235d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:54.997469 6422 ts_tablet_manager.cc:1403] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.327s user 0.010s sys 0.015s
I20260620 19:57:55.002339 6428 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.023238 6428 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.024067 6296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e39bd0f2de444d2acf683f9f18716da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.025210 6428 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.027153 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:57:55.029150 6195 leader_election.cc:343] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.031049 6367 tablet_service.cc:1511] Processing CreateTablet for tablet 779819d94546427fa367040185e417ab (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260620 19:57:55.032356 6367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779819d94546427fa367040185e417ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.033589 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.034255 6381 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.036603 6422 raft_consensus.cc:359] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:55.037317 6422 raft_consensus.cc:385] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.037612 6422 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.038414 6422 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:55.041208 6422 ts_tablet_manager.cc:1434] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:57:55.035578 6368 tablet_service.cc:1511] Processing CreateTablet for tablet 99e667a051414b87955b78b2520409b5 (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260620 19:57:55.043399 6368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e667a051414b87955b78b2520409b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:55.051488 6198 consensus_peers.cc:597] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Couldn't send request to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:57:55.060264 6428 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:55.047415 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c91ae8424d495f92c57a6fcfc235d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.068005 6220 tablet_service.cc:1511] Processing CreateTablet for tablet 1e39bd0f2de444d2acf683f9f18716da (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260620 19:57:55.066270 6433 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260620 19:57:55.076459 6198 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:55.076877 6429 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:55.079654 6221 tablet_service.cc:1511] Processing CreateTablet for tablet 779819d94546427fa367040185e417ab (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260620 19:57:55.081235 6220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e39bd0f2de444d2acf683f9f18716da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.080199 6428 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.083139 6428 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.083567 6428 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.090672 6421 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.091214 6421 ts_tablet_manager.cc:1403] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.341s user 0.017s sys 0.008s
I20260620 19:57:55.092015 6222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d4b5cc9261e42c381b94cd3961f1cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.094064 6421 raft_consensus.cc:359] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.094873 6421 raft_consensus.cc:385] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.095211 6421 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.095963 6421 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.069887 6293 tablet_service.cc:1511] Processing CreateTablet for tablet 779819d94546427fa367040185e417ab (DEFAULT_TABLE table=test-workload [id=bdc4fe719f514b06a7419a6a405069a5]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260620 19:57:55.109150 6293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779819d94546427fa367040185e417ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:57:55.109189 6423 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.109766 6423 ts_tablet_manager.cc:1403] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.369s user 0.015s sys 0.007s
I20260620 19:57:55.110394 6428 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.113157 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:55.113487 6221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779819d94546427fa367040185e417ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 19:57:55.114514 6195 leader_election.cc:343] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.113397 6423 raft_consensus.cc:359] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.115923 6423 raft_consensus.cc:385] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.116276 6423 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.114449 6422 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.118172 6421 ts_tablet_manager.cc:1434] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260620 19:57:55.119311 6421 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.118872 6423 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.123193 6423 ts_tablet_manager.cc:1434] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.013s user 0.007s sys 0.002s
I20260620 19:57:55.129057 6422 tablet_bootstrap.cc:654] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.137619 6428 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.137797 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:55.138391 6380 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.143424 6423 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.170298 6421 tablet_bootstrap.cc:654] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.177356 6380 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.178539 6198 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:55.180147 6450 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.180979 6423 tablet_bootstrap.cc:654] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.187177 6450 raft_consensus.cc:697] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:55.205276 6450 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.222990 6422 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.223804 6422 ts_tablet_manager.cc:1403] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.110s user 0.023s sys 0.022s
I20260620 19:57:55.222786 6423 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.225467 6423 ts_tablet_manager.cc:1403] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.006s
I20260620 19:57:55.226727 6422 raft_consensus.cc:359] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.227495 6422 raft_consensus.cc:385] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.227789 6422 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.228309 6423 raft_consensus.cc:359] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.228929 6423 raft_consensus.cc:385] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.228505 6422 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.229980 6421 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.230358 6421 ts_tablet_manager.cc:1403] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.017s
I20260620 19:57:55.229219 6423 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.231676 6422 ts_tablet_manager.cc:1434] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260620 19:57:55.232805 6422 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.231323 6423 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.233623 6421 raft_consensus.cc:359] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.234329 6421 raft_consensus.cc:385] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.234642 6421 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.235455 6423 ts_tablet_manager.cc:1434] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20260620 19:57:55.235378 6421 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.236392 6423 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.238268 6128 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:55.243368 6422 tablet_bootstrap.cc:654] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.244681 6421 ts_tablet_manager.cc:1434] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260620 19:57:55.250378 6423 tablet_bootstrap.cc:654] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.252952 6421 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.258656 6429 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.259119 6429 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.261150 6429 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.262246 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.262727 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:55.264075 6273 leader_election.cc:343] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:57:55.264554 6271 leader_election.cc:343] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.264966 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:55.266166 6429 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:55.268534 6421 tablet_bootstrap.cc:654] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.273488 6422 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.273859 6421 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.273943 6423 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.274247 6421 ts_tablet_manager.cc:1403] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.002s
I20260620 19:57:55.273941 6422 ts_tablet_manager.cc:1403] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.010s
I20260620 19:57:55.274585 6423 ts_tablet_manager.cc:1403] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.018s
I20260620 19:57:55.276954 6421 raft_consensus.cc:359] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.277480 6421 raft_consensus.cc:385] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.277751 6421 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.277616 6423 raft_consensus.cc:359] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.277743 6422 raft_consensus.cc:359] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.278429 6423 raft_consensus.cc:385] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.278716 6422 raft_consensus.cc:385] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.278755 6423 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.278683 6421 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.279248 6422 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.279819 6423 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.280066 6422 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.280794 6421 ts_tablet_manager.cc:1434] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 19:57:55.281752 6421 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.282546 6422 ts_tablet_manager.cc:1434] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.008s user 0.000s sys 0.004s
I20260620 19:57:55.288084 6421 tablet_bootstrap.cc:654] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.288352 6423 ts_tablet_manager.cc:1434] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260620 19:57:55.289315 6423 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.294322 6422 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.296274 6423 tablet_bootstrap.cc:654] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.301728 6422 tablet_bootstrap.cc:654] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.303915 6421 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.304394 6421 ts_tablet_manager.cc:1403] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.006s
I20260620 19:57:55.307121 6423 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.307539 6423 ts_tablet_manager.cc:1403] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20260620 19:57:55.307482 6421 raft_consensus.cc:359] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.308362 6421 raft_consensus.cc:385] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.308414 6422 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.308811 6421 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.309015 6422 ts_tablet_manager.cc:1403] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260620 19:57:55.309605 6421 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.310057 6423 raft_consensus.cc:359] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.310719 6423 raft_consensus.cc:385] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.310968 6423 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.310925 6450 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:55.311456 6422 raft_consensus.cc:359] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.312224 6422 raft_consensus.cc:385] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.311877 6423 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.314031 6423 ts_tablet_manager.cc:1434] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 19:57:55.314853 6423 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.316560 6421 ts_tablet_manager.cc:1434] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260620 19:57:55.317552 6421 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.312489 6422 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.318315 6422 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.320248 6422 ts_tablet_manager.cc:1434] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260620 19:57:55.321386 6423 tablet_bootstrap.cc:654] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.323307 6450 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.323787 6450 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.324062 6429 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.324657 6429 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.327363 6429 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.327749 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.328401 6380 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.329456 6450 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.329480 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.329773 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.330173 6307 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.330353 6380 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:55.330636 6422 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.330794 6423 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.330121 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:55.331226 6423 ts_tablet_manager.cc:1403] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260620 19:57:55.331346 6232 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:55.332564 6273 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:55.333191 6198 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.334280 6423 raft_consensus.cc:359] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:55.334956 6423 raft_consensus.cc:385] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.334851 6429 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.335318 6423 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.335454 6429 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.335916 6429 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.336154 6423 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:55.337256 6422 tablet_bootstrap.cc:654] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.339046 6450 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.339421 6450 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.339732 6450 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.341614 6423 ts_tablet_manager.cc:1434] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260620 19:57:55.342494 6423 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.347920 6423 tablet_bootstrap.cc:654] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.351316 6474 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.351944 6474 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:55.355371 6421 tablet_bootstrap.cc:654] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.356015 6429 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.358459 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.359268 6307 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.356410 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.360600 6380 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.362632 6198 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.362703 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:55.354565 6474 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.365797 6474 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:55.365996 6429 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.366860 6422 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.367354 6422 ts_tablet_manager.cc:1403] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.014s
I20260620 19:57:55.371906 6422 raft_consensus.cc:359] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.372807 6422 raft_consensus.cc:385] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.373150 6422 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.363843 6428 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.375214 6422 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.366525 6232 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.377235 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:55.377969 6380 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.380980 6450 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.374750 6428 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.382889 6428 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.383548 6450 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.384112 6450 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.384562 6450 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.387172 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:55.387830 6379 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.390110 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:55.390551 6380 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:55.390691 6306 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.391916 6273 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:55.392731 6429 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.393189 6429 raft_consensus.cc:697] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:55.393966 6429 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.401916 6232 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:55.403292 6422 ts_tablet_manager.cc:1434] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.036s user 0.008s sys 0.001s
I20260620 19:57:55.404054 6422 tablet_bootstrap.cc:492] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.414942 6423 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.415545 6423 ts_tablet_manager.cc:1403] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.073s user 0.018s sys 0.015s
I20260620 19:57:55.414968 6421 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.417932 6421 ts_tablet_manager.cc:1403] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.101s user 0.013s sys 0.002s
I20260620 19:57:55.426916 6379 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.427392 6380 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.417044 6306 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.433034 6423 raft_consensus.cc:359] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.438874 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.441387 6423 raft_consensus.cc:385] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.442672 6423 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.441936 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.439754 6450 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.444422 6305 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
W20260620 19:57:55.444725 6198 leader_election.cc:343] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.445276 6423 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.445816 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:55.447552 6450 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.448019 6450 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.448427 6450 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.453063 6198 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.453111 6428 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:55.455148 6488 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.455281 6428 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.456415 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:55.457036 6306 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.461922 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:55.462049 6421 raft_consensus.cc:359] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.462968 6379 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.465755 6488 raft_consensus.cc:697] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:55.463068 6421 raft_consensus.cc:385] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.466445 6421 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.466638 6488 consensus_queue.cc:237] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.468102 6421 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.470532 6421 ts_tablet_manager.cc:1434] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.052s user 0.004s sys 0.019s
I20260620 19:57:55.471510 6421 tablet_bootstrap.cc:492] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.471875 6423 ts_tablet_manager.cc:1434] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.056s user 0.006s sys 0.005s
I20260620 19:57:55.472906 6423 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.480067 6422 tablet_bootstrap.cc:654] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.484153 6306 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.485237 6195 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.482899 6379 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.486766 6492 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.488914 6492 raft_consensus.cc:697] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:55.489715 6492 consensus_queue.cc:237] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:57:55.491454 6423 tablet_bootstrap.cc:654] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.494850 6421 tablet_bootstrap.cc:654] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.505390 6450 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.508716 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:55.509486 6306 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.509727 6130 catalog_manager.cc:5696] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:55.517235 6492 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.517877 6492 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.521474 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.522763 6307 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.524564 6422 tablet_bootstrap.cc:492] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.524396 6195 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.525138 6422 ts_tablet_manager.cc:1403] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.017s
I20260620 19:57:55.525584 6488 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.526099 6488 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.526551 6488 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.538146 6129 catalog_manager.cc:5696] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:55.539578 6306 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.541368 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.542543 6493 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.543212 6493 raft_consensus.cc:697] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:55.544224 6493 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.551256 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:57:55.552896 6198 leader_election.cc:343] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.553836 6450 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.556991 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.557839 6380 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.560676 6492 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.562887 6488 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.566238 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:55.566807 6307 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.573541 6422 raft_consensus.cc:359] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.576236 6307 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.577966 6423 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.577929 6195 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.578478 6423 ts_tablet_manager.cc:1403] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.106s user 0.029s sys 0.045s
I20260620 19:57:55.579106 6492 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.579602 6492 raft_consensus.cc:697] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:55.574440 6422 raft_consensus.cc:385] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.580374 6422 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.581228 6422 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.581425 6423 raft_consensus.cc:359] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.582330 6423 raft_consensus.cc:385] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.582650 6423 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.583390 6423 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.585486 6423 ts_tablet_manager.cc:1434] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:57:55.586418 6422 ts_tablet_manager.cc:1434] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.061s user 0.010s sys 0.000s
I20260620 19:57:55.586584 6423 tablet_bootstrap.cc:492] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.587697 6422 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.590006 6495 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.590456 6495 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.591449 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:55.591993 6380 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.592947 6495 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.580376 6492 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.597870 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:55.599000 6488 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:57:55.599562 6345 leader_election.cc:343] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.599936 6422 tablet_bootstrap.cc:654] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.601183 6380 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:55.602035 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.606228 6307 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:55.607858 6344 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:55.607496 6128 catalog_manager.cc:5696] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:55.609604 6495 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.609975 6495 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.610344 6495 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.610126 6128 catalog_manager.cc:5696] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:55.618546 6495 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.622123 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:57:55.623518 6345 leader_election.cc:343] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.624150 6421 tablet_bootstrap.cc:492] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.625871 6421 ts_tablet_manager.cc:1403] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.155s user 0.035s sys 0.064s
I20260620 19:57:55.626132 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:55.626735 6307 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.628635 6421 raft_consensus.cc:359] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.629305 6421 raft_consensus.cc:385] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.629570 6421 raft_consensus.cc:740] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.630190 6421 consensus_queue.cc:260] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.632100 6421 ts_tablet_manager.cc:1434] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260620 19:57:55.633024 6307 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:55.633664 6421 tablet_bootstrap.cc:492] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.624325 6423 tablet_bootstrap.cc:654] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.625003 6495 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.633766 6128 catalog_manager.cc:5696] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:55.635298 6344 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:55.636919 6495 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.637359 6495 raft_consensus.cc:697] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:55.638504 6495 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.645802 6421 tablet_bootstrap.cc:654] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.652671 6306 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.653357 6422 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.656993 6422 ts_tablet_manager.cc:1403] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.005s
I20260620 19:57:55.661131 6421 tablet_bootstrap.cc:492] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.661561 6421 ts_tablet_manager.cc:1403] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.013s
I20260620 19:57:55.662397 6488 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:55.664263 6423 tablet_bootstrap.cc:492] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.663985 6421 raft_consensus.cc:359] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.662958 6128 catalog_manager.cc:5696] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:55.664664 6423 ts_tablet_manager.cc:1403] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.078s user 0.017s sys 0.013s
I20260620 19:57:55.664625 6421 raft_consensus.cc:385] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.665040 6421 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.665681 6421 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.667328 6423 raft_consensus.cc:359] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.667778 6421 ts_tablet_manager.cc:1434] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260620 19:57:55.667984 6423 raft_consensus.cc:385] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.668231 6423 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.668406 6488 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:55.668604 6421 tablet_bootstrap.cc:492] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.668850 6423 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.670873 6423 ts_tablet_manager.cc:1434] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:57:55.672714 6423 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.676281 6422 raft_consensus.cc:359] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.677241 6422 raft_consensus.cc:385] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.677563 6422 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.678328 6422 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.683257 6422 ts_tablet_manager.cc:1434] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.026s user 0.007s sys 0.001s
I20260620 19:57:55.684146 6422 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.689954 6422 tablet_bootstrap.cc:654] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.701028 6421 tablet_bootstrap.cc:654] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.705044 6423 tablet_bootstrap.cc:654] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.712414 6422 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.712999 6422 ts_tablet_manager.cc:1403] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.004s
I20260620 19:57:55.716190 6422 raft_consensus.cc:359] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.717039 6422 raft_consensus.cc:385] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.717334 6422 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.719269 6422 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.721056 6422 ts_tablet_manager.cc:1434] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 19:57:55.721946 6422 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.726560 6422 tablet_bootstrap.cc:654] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.728330 6423 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.728835 6423 ts_tablet_manager.cc:1403] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.012s
I20260620 19:57:55.732611 6423 raft_consensus.cc:359] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.733695 6423 raft_consensus.cc:385] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.734215 6423 raft_consensus.cc:740] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.735270 6423 consensus_queue.cc:260] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.738359 6423 ts_tablet_manager.cc:1434] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260620 19:57:55.739567 6423 tablet_bootstrap.cc:492] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.744973 6422 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.745428 6422 ts_tablet_manager.cc:1403] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.007s
I20260620 19:57:55.748097 6422 raft_consensus.cc:359] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.748749 6422 raft_consensus.cc:385] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.749010 6422 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.749846 6423 tablet_bootstrap.cc:654] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.749706 6422 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.752461 6422 ts_tablet_manager.cc:1434] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260620 19:57:55.753492 6422 tablet_bootstrap.cc:492] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.762394 6422 tablet_bootstrap.cc:654] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.764101 6421 tablet_bootstrap.cc:492] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.764609 6421 ts_tablet_manager.cc:1403] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.096s user 0.020s sys 0.039s
I20260620 19:57:55.766492 6421 raft_consensus.cc:359] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.767043 6421 raft_consensus.cc:385] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.767269 6421 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.767845 6423 tablet_bootstrap.cc:492] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.768384 6423 ts_tablet_manager.cc:1403] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.014s
I20260620 19:57:55.767858 6421 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.769896 6422 tablet_bootstrap.cc:492] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.770300 6422 ts_tablet_manager.cc:1403] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.006s
I20260620 19:57:55.770445 6421 ts_tablet_manager.cc:1434] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:57:55.770809 6423 raft_consensus.cc:359] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.771430 6423 raft_consensus.cc:385] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.771696 6423 raft_consensus.cc:740] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.772189 6421 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.772464 6423 consensus_queue.cc:260] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.773725 6422 raft_consensus.cc:359] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.774380 6422 raft_consensus.cc:385] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.774672 6422 raft_consensus.cc:740] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.775411 6422 consensus_queue.cc:260] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.777724 6422 ts_tablet_manager.cc:1434] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:55.778702 6422 tablet_bootstrap.cc:492] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.778812 6421 tablet_bootstrap.cc:654] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.783202 6423 ts_tablet_manager.cc:1434] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260620 19:57:55.787578 6423 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.792081 6423 tablet_bootstrap.cc:654] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.792979 6421 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.793421 6421 ts_tablet_manager.cc:1403] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.011s
I20260620 19:57:55.793752 6422 tablet_bootstrap.cc:654] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.795950 6421 raft_consensus.cc:359] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.796671 6421 raft_consensus.cc:385] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.796981 6423 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.796933 6421 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.797814 6423 ts_tablet_manager.cc:1403] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 19:57:55.798211 6422 tablet_bootstrap.cc:492] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.798025 6421 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.798621 6422 ts_tablet_manager.cc:1403] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.003s
I20260620 19:57:55.800482 6421 ts_tablet_manager.cc:1434] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:57:55.800757 6423 raft_consensus.cc:359] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.801427 6423 raft_consensus.cc:385] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.801430 6421 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.801821 6423 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.801452 6422 raft_consensus.cc:359] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.802291 6422 raft_consensus.cc:385] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.802546 6422 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.802634 6423 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.805028 6423 ts_tablet_manager.cc:1434] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:57:55.806744 6423 tablet_bootstrap.cc:492] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.806924 6421 tablet_bootstrap.cc:654] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.803315 6422 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.809340 6422 ts_tablet_manager.cc:1434] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260620 19:57:55.809770 6429 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.810245 6429 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.811071 6422 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.812448 6429 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.814740 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:55.815615 6232 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:55.817191 6271 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.814874 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.819350 6429 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.819830 6429 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.820266 6429 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:57:55.823356 6273 leader_election.cc:343] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.831674 6433 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:55.834028 6421 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.835340 6423 tablet_bootstrap.cc:654] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.834491 6421 ts_tablet_manager.cc:1403] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.009s
I20260620 19:57:55.840502 6433 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.840849 6421 raft_consensus.cc:359] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.841081 6433 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.841548 6421 raft_consensus.cc:385] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.841840 6421 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.842765 6421 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.848769 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.849269 6423 tablet_bootstrap.cc:492] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.849382 6421 ts_tablet_manager.cc:1434] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260620 19:57:55.849732 6423 ts_tablet_manager.cc:1403] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.004s
W20260620 19:57:55.850075 6273 leader_election.cc:343] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.850282 6421 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.847880 6433 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.852715 6423 raft_consensus.cc:359] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.853382 6423 raft_consensus.cc:385] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.854307 6423 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.855172 6423 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.862910 6423 ts_tablet_manager.cc:1434] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.013s user 0.004s sys 0.007s
I20260620 19:57:55.863766 6423 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.865772 6421 tablet_bootstrap.cc:654] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.868434 6422 tablet_bootstrap.cc:654] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.868636 6423 tablet_bootstrap.cc:654] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:55.869073 6523 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:55.868955 6429 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.882194 6429 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.882419 6488 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.883004 6488 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.884554 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:57:55.886205 6273 leader_election.cc:343] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.886353 6488 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.887619 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.888321 6306 raft_consensus.cc:2393] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:55.889139 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:55.889832 6231 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.890518 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:55.895038 6271 leader_election.cc:343] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.895283 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.895507 6271 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:55.895591 6495 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.896513 6421 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:55.896834 6422 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:55.896957 6421 ts_tablet_manager.cc:1403] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.016s
I20260620 19:57:55.896606 6495 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.897089 6433 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:55.897233 6422 ts_tablet_manager.cc:1403] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.086s user 0.020s sys 0.012s
I20260620 19:57:55.898669 6495 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.900039 6422 raft_consensus.cc:359] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.900424 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.900705 6422 raft_consensus.cc:385] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.901050 6422 raft_consensus.cc:740] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.901289 6423 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:55.901677 6423 ts_tablet_manager.cc:1403] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.004s
W20260620 19:57:55.901670 6344 leader_election.cc:343] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:55.901638 6422 consensus_queue.cc:260] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.903232 6422 ts_tablet_manager.cc:1434] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:57:55.903455 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:55.904086 6232 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:55.904433 6422 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:55.904119 6423 raft_consensus.cc:359] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.904811 6423 raft_consensus.cc:385] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.905069 6423 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.900038 6421 raft_consensus.cc:359] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.906093 6421 raft_consensus.cc:385] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:55.906332 6421 raft_consensus.cc:740] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:55.907990 6423 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.909024 6421 consensus_queue.cc:260] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.908275 6345 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:55.911134 6427 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.911265 6421 ts_tablet_manager.cc:1434] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260620 19:57:55.911741 6427 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.913144 6421 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:55.912671 6427 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.928982 6488 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.929484 6488 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.930733 6495 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.931118 6488 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.931191 6495 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.931998 6492 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:55.935245 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:55.936013 6232 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:55.936291 6523 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.936523 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.936743 6423 ts_tablet_manager.cc:1434] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.035s user 0.006s sys 0.002s
I20260620 19:57:55.919016 6429 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.936780 6523 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.937206 6345 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.938057 6423 tablet_bootstrap.cc:492] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:55.931712 6488 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.942289 6306 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.942309 6488 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.943379 6523 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.937454 6429 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.944514 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.945319 6379 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:55.946691 6429 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.919253 6198 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:55.947763 6488 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.948235 6488 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.951335 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:55.952129 6232 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:55.952373 6427 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.953790 6273 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:55.955226 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.956389 6495 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.956681 6231 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:55.957015 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.957047 6495 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.957659 6495 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.959089 6379 raft_consensus.cc:2393] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:55.959174 6495 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.960099 6523 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.960400 6492 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.961069 6492 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.961735 6195 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:55.962517 6493 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:55.962893 6493 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.963230 6493 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.963587 6492 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:55.960489 6523 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:55.964331 6523 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:57:55.964847 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:57:55.958338 6429 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.974843 6427 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.982872 6427 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.981559 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.983129 6529 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:55.983471 6427 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.983611 6529 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.985596 6529 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.985868 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:55.984599 6271 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:55.986701 6427 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:55.986488 6305 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:55.973462 6429 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.976529 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:55.991184 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:55.990581 6433 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:55.992386 6433 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:55.993533 6433 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:55.994316 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.994956 6378 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:55.995278 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:55.994256 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:56.005981 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:55.999325 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.000162 6231 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:55.998257 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.000960 6381 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:56.003733 6304 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
W20260620 19:57:56.022877 6344 leader_election.cc:343] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.027209 6493 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:57:56.011096 6195 leader_election.cc:343] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.020701 6198 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.036103 6433 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.039031 6493 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.039867 6433 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.040764 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.041105 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.041353 6232 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:56.041702 6231 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:56.042964 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:57:56.044456 6273 leader_election.cc:343] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.044970 6433 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.045349 6495 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.046067 6422 tablet_bootstrap.cc:654] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.046506 6379 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:56.046483 6305 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.052865 6493 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.053316 6493 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.053705 6493 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.054119 6428 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.055823 6198 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:56.058871 6428 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.059201 6428 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.059407 6428 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.060382 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:56.061138 6379 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:56.061517 6423 tablet_bootstrap.cc:654] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.062796 6429 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.063788 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:56.063894 6344 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.064396 6379 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:56.065415 6527 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.064438 6271 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:56.065958 6527 raft_consensus.cc:697] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:56.065897 6273 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:56.063896 6128 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:56.068945 6421 tablet_bootstrap.cc:654] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.073733 6433 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.074106 6433 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.074430 6433 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.080384 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.080888 6428 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.081537 6527 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.076699 6429 raft_consensus.cc:2764] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 19:57:56.085922 6523 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.087121 6306 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:56.077275 6488 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.088814 6492 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260620 19:57:56.095641 6433 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.100353 6495 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.101044 6345 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:56.104533 6427 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.104894 6427 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.105216 6427 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.105870 6195 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:56.106889 6492 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.106889 6345 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:56.107678 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.108165 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:56.109145 6523 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.114588 6428 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.114666 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.115478 6495 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.118258 6427 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.119481 6433 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.113705 6378 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:56.120523 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:57:56.124563 6195 leader_election.cc:343] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.121019 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.112553 6527 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.118304 6493 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.125455 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.126701 6306 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:56.127043 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.126989 6450 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.127528 6492 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.129861 6428 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:56.135262 6231 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:56.131927 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.134758 6423 tablet_bootstrap.cc:492] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:56.128594 6527 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.134485 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.130530 6493 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.136374 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.143832 6423 ts_tablet_manager.cc:1403] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.206s user 0.015s sys 0.003s
W20260620 19:57:56.181037 6271 leader_election.cc:343] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.181511 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.182283 6523 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.177366 6305 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:56.164508 6427 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.162994 6232 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:56.187134 6527 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.191136 6422 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:56.191742 6422 ts_tablet_manager.cc:1403] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.287s user 0.017s sys 0.032s
I20260620 19:57:56.186409 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:56.203291 6376 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:56.177182 6428 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260620 19:57:56.204105 6422 raft_consensus.cc:359] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.204818 6422 raft_consensus.cc:385] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.205000 6422 raft_consensus.cc:740] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.205538 6422 consensus_queue.cc:260] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.177182 6345 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.194550 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.170586 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.215526 6229 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:56.208513 6527 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.208757 6495 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.218954 6422 ts_tablet_manager.cc:1434] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260620 19:57:56.219734 6422 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:56.218838 6379 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:56.217756 6427 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.225488 6230 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:56.226856 6423 raft_consensus.cc:359] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.228736 6423 raft_consensus.cc:385] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.229023 6423 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.230822 6421 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.231190 6421 ts_tablet_manager.cc:1403] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.319s user 0.006s sys 0.019s
I20260620 19:57:56.209708 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:56.228310 6427 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.233673 6423 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.236368 6423 ts_tablet_manager.cc:1434] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.056s user 0.006s sys 0.000s
I20260620 19:57:56.230235 6527 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.237329 6423 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:56.223245 6198 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:56.239303 6492 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.240218 6271 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.243217 6433 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.244714 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.246856 6229 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:56.246088 6381 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.258721 6527 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:56.262313 6273 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.266279 6433 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.268626 6198 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:56.270960 6450 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.272035 6304 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.275995 6423 tablet_bootstrap.cc:654] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.288065 6421 raft_consensus.cc:359] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.288808 6421 raft_consensus.cc:385] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.289055 6421 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.289351 6422 tablet_bootstrap.cc:654] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.289712 6421 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.276341 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.295670 6421 ts_tablet_manager.cc:1434] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.064s user 0.005s sys 0.002s
I20260620 19:57:56.296478 6421 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.302403 6307 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:56.302773 6495 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.310146 6130 catalog_manager.cc:5696] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:56.310889 6495 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.315222 6344 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.315630 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.317386 6529 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.317512 6428 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260620 19:57:56.316412 6427 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.325184 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.326004 6228 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:56.319650 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.331482 6433 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.332696 6433 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.329712 6345 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.334898 6529 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.335258 6529 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.335579 6529 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.340287 6433 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.328765 6495 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.342581 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:57:56.343987 6273 leader_election.cc:343] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.344303 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:56.346328 6271 leader_election.cc:343] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.346771 6271 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.347854 6433 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.328338 6304 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.358008 6529 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.362519 6421 tablet_bootstrap.cc:654] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.363878 6529 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.368948 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.369707 6230 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.376132 6421 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.376581 6421 ts_tablet_manager.cc:1403] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.005s
I20260620 19:57:56.378149 6230 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.380355 6345 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.382720 6544 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.383235 6544 raft_consensus.cc:697] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:57:56.382316 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 67145.1us
I20260620 19:57:56.379139 6421 raft_consensus.cc:359] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.384073 6544 consensus_queue.cc:237] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.388527 6422 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:56.388845 6421 raft_consensus.cc:385] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.389063 6421 raft_consensus.cc:740] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.389370 6422 ts_tablet_manager.cc:1403] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.170s user 0.024s sys 0.022s
I20260620 19:57:56.390427 6421 consensus_queue.cc:260] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.391957 6422 raft_consensus.cc:359] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.392606 6422 raft_consensus.cc:385] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.392861 6422 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.393467 6422 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.394502 6421 ts_tablet_manager.cc:1434] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260620 19:57:56.396696 6421 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.400602 6422 ts_tablet_manager.cc:1434] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260620 19:57:56.401320 6422 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:56.402374 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.403240 6303 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:56.411862 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.412290 6421 tablet_bootstrap.cc:654] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.412623 6303 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.416011 6422 tablet_bootstrap.cc:654] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.416574 6381 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.445899 6344 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.447093 6527 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.447501 6344 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.453841 6527 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.454304 6527 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:56.455091 6527 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.456756 6423 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:56.456569 6433 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.458401 6303 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.458452 6427 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.459740 6423 ts_tablet_manager.cc:1403] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.223s user 0.030s sys 0.040s
I20260620 19:57:56.459818 6427 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.462787 6422 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:56.462530 6423 raft_consensus.cc:359] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.463048 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.463249 6422 ts_tablet_manager.cc:1403] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.005s
I20260620 19:57:56.464318 6427 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.465677 6433 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.467348 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.468309 6230 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.469570 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.470569 6422 raft_consensus.cc:359] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.471215 6422 raft_consensus.cc:385] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.471463 6422 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.472122 6422 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.475884 6422 ts_tablet_manager.cc:1434] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260620 19:57:56.477013 6422 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:56.477056 6423 raft_consensus.cc:385] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.477440 6423 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.478071 6423 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.480058 6423 ts_tablet_manager.cc:1434] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260620 19:57:56.481709 6544 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:57:56.482082 6544 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:57:56.483155 6421 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.483610 6544 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:56.483932 6421 ts_tablet_manager.cc:1403] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.087s user 0.016s sys 0.007s
I20260620 19:57:56.486614 6130 catalog_manager.cc:5696] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:56.489763 6423 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:56.490420 6422 tablet_bootstrap.cc:654] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.486527 6421 raft_consensus.cc:359] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.493808 6421 raft_consensus.cc:385] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.494211 6421 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.495039 6421 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.494336 6492 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.497188 6421 ts_tablet_manager.cc:1434] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260620 19:57:56.497153 6492 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.498450 6421 tablet_bootstrap.cc:492] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.500021 6492 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.503322 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.503616 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:56.503980 6303 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:56.504184 6376 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:56.505374 6198 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.506850 6423 tablet_bootstrap.cc:654] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.507254 6492 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.508042 6492 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.508661 6492 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.506112 6544 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.515811 6421 tablet_bootstrap.cc:654] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.517369 6544 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.517879 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.524425 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.529241 6421 tablet_bootstrap.cc:492] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.529770 6421 ts_tablet_manager.cc:1403] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.006s
I20260620 19:57:56.530040 6344 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.531117 6544 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:57:56.534330 6421 raft_consensus.cc:359] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.534993 6421 raft_consensus.cc:385] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.535259 6421 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.536064 6421 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.538877 6421 ts_tablet_manager.cc:1434] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260620 19:57:56.539945 6421 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.546913 6428 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.547575 6428 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.549242 6421 tablet_bootstrap.cc:654] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.551716 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:56.552353 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.552557 6376 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
W20260620 19:57:56.554107 6195 leader_election.cc:343] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.554931 6198 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:56.550212 6428 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.556001 6450 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.556483 6450 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.556910 6450 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.571905 6492 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.577085 6381 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:57:56.580580 6523 consensus_queue.cc:1059] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:56.582039 6429 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:57:56.582507 6429 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:56.583611 6429 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.584708 6429 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:56.589082 6229 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.596661 6492 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.578296 6450 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.600613 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:56.601415 6381 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.610222 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:56.611023 6376 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.617705 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.619859 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.620661 6303 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:57:56.629498 6195 leader_election.cc:343] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.631771 6381 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:56.634030 6544 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:56.639488 6376 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:56.642292 6421 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.642529 6423 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:56.642758 6421 ts_tablet_manager.cc:1403] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.103s user 0.048s sys 0.040s
I20260620 19:57:56.643383 6198 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:56.644405 6428 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.644920 6428 raft_consensus.cc:697] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:56.646067 6423 ts_tablet_manager.cc:1403] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.156s user 0.072s sys 0.057s
I20260620 19:57:56.645654 6428 consensus_queue.cc:237] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.646278 6421 raft_consensus.cc:359] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.646937 6421 raft_consensus.cc:385] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.647192 6421 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.647804 6421 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.648487 6423 raft_consensus.cc:359] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.649087 6423 raft_consensus.cc:385] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.649341 6423 raft_consensus.cc:740] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.650449 6421 ts_tablet_manager.cc:1434] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:56.650053 6423 consensus_queue.cc:260] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.649745 6128 catalog_manager.cc:5696] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:56.651302 6421 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.651355 6429 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.651907 6429 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.652311 6423 ts_tablet_manager.cc:1434] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:57:56.653124 6423 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:56.653836 6429 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.657231 6421 tablet_bootstrap.cc:654] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.659009 6423 tablet_bootstrap.cc:654] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.660264 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:56.661015 6376 raft_consensus.cc:2393] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:56.668743 6198 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.669988 6493 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.670547 6493 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:56.671604 6493 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.677137 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.679435 6422 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:56.680035 6422 ts_tablet_manager.cc:1403] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.203s user 0.026s sys 0.014s
I20260620 19:57:56.680351 6271 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.681933 6429 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.696341 6450 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.700006 6429 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.703147 6303 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:56.704388 6422 raft_consensus.cc:359] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.710660 6422 raft_consensus.cc:385] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.710950 6422 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.711678 6422 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.708381 6421 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.708525 6527 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.713712 6422 ts_tablet_manager.cc:1434] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260620 19:57:56.714408 6421 ts_tablet_manager.cc:1403] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.063s user 0.025s sys 0.027s
I20260620 19:57:56.714512 6422 tablet_bootstrap.cc:492] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:56.717278 6421 raft_consensus.cc:359] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.717916 6421 raft_consensus.cc:385] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.718179 6421 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.718961 6421 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.720422 6130 catalog_manager.cc:5696] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:56.722277 6422 tablet_bootstrap.cc:654] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.725292 6421 ts_tablet_manager.cc:1434] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260620 19:57:56.726689 6421 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.738266 6564 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.738744 6564 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.740816 6564 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.741493 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.741869 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:56.742491 6381 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
W20260620 19:57:56.742666 6195 leader_election.cc:343] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.743796 6423 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:56.744938 6423 ts_tablet_manager.cc:1403] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.092s user 0.035s sys 0.036s
I20260620 19:57:56.743687 6198 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:56.746701 6421 tablet_bootstrap.cc:654] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.747833 6423 raft_consensus.cc:359] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.748891 6423 raft_consensus.cc:385] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.749150 6423 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.749504 6564 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.749850 6564 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.749773 6423 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.750245 6564 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.752204 6421 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.752667 6421 ts_tablet_manager.cc:1403] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20260620 19:57:56.752530 6130 catalog_manager.cc:5696] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:56.754062 6422 tablet_bootstrap.cc:492] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:56.754470 6422 ts_tablet_manager.cc:1403] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20260620 19:57:56.755967 6421 raft_consensus.cc:359] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.756631 6421 raft_consensus.cc:385] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.756772 6422 raft_consensus.cc:359] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.756625 6423 ts_tablet_manager.cc:1434] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260620 19:57:56.757156 6421 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.757576 6422 raft_consensus.cc:385] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.757902 6422 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.757261 6564 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.758095 6421 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.760159 6421 ts_tablet_manager.cc:1434] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260620 19:57:56.761795 6421 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.762005 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:56.762987 6381 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.758520 6422 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.766312 6422 ts_tablet_manager.cc:1434] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260620 19:57:56.760241 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.767141 6422 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:56.767488 6421 tablet_bootstrap.cc:654] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
W20260620 19:57:56.767747 6195 leader_election.cc:343] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.761010 6423 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:56.774891 6422 tablet_bootstrap.cc:654] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.781095 6423 tablet_bootstrap.cc:654] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.782148 6527 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.782786 6527 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.786939 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.787745 6229 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:56.790990 6345 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.792061 6495 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.792505 6495 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.792933 6495 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.793927 6381 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:56.799558 6564 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.801055 6198 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:56.802099 6492 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.804178 6492 raft_consensus.cc:697] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:56.805274 6492 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.809108 6527 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.809293 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.811554 6421 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.809880 6303 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:56.812093 6421 ts_tablet_manager.cc:1403] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.050s user 0.016s sys 0.029s
I20260620 19:57:56.814092 6421 raft_consensus.cc:359] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.814570 6421 raft_consensus.cc:385] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.814853 6421 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.815726 6421 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.817852 6421 ts_tablet_manager.cc:1434] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:57:56.818542 6423 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:56.818811 6421 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.819022 6423 ts_tablet_manager.cc:1403] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.022s
I20260620 19:57:56.819195 6495 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.822000 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.821931 6423 raft_consensus.cc:359] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.822770 6423 raft_consensus.cc:385] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.823128 6229 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.824548 6423 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.825182 6421 tablet_bootstrap.cc:654] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.825313 6423 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.826205 6523 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.826668 6523 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.828751 6523 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:56.830166 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:57:56.831812 6273 leader_election.cc:343] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.832559 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:56.833231 6423 ts_tablet_manager.cc:1434] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
W20260620 19:57:56.833911 6271 leader_election.cc:343] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.834113 6423 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:56.834327 6271 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.835829 6523 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:56.837451 6527 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:56.838043 6527 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.843104 6529 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.845669 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:56.848474 6345 leader_election.cc:343] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:56.849364 6422 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:56.851099 6422 ts_tablet_manager.cc:1403] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.012s
I20260620 19:57:56.853847 6422 raft_consensus.cc:359] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.854460 6422 raft_consensus.cc:385] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.854729 6422 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.855046 6423 tablet_bootstrap.cc:654] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.855451 6422 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.861205 6422 ts_tablet_manager.cc:1434] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260620 19:57:56.862063 6422 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:56.867679 6422 tablet_bootstrap.cc:654] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.868026 6130 catalog_manager.cc:5696] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:56.871016 6527 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.872356 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:56.875691 6544 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.875906 6304 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:56.876972 6229 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.877252 6344 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:56.878149 6544 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:56.879048 6544 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:56.879379 6544 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.880914 6421 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.881326 6421 ts_tablet_manager.cc:1403] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.063s user 0.025s sys 0.026s
I20260620 19:57:56.880939 6423 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:56.881933 6423 ts_tablet_manager.cc:1403] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.000s
I20260620 19:57:56.878196 6345 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:56.883349 6427 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.883793 6427 raft_consensus.cc:697] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:56.884248 6421 raft_consensus.cc:359] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.884929 6421 raft_consensus.cc:385] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.884557 6427 consensus_queue.cc:237] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.885190 6421 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.885841 6421 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.886114 6423 raft_consensus.cc:359] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.886705 6423 raft_consensus.cc:385] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.886973 6423 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.887601 6423 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.888099 6495 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.889528 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.890115 6304 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.891014 6423 ts_tablet_manager.cc:1434] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260620 19:57:56.891873 6423 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:56.895475 6421 ts_tablet_manager.cc:1434] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260620 19:57:56.896312 6421 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:56.900913 6422 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:56.901335 6422 ts_tablet_manager.cc:1403] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.000s
I20260620 19:57:56.903430 6304 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.904664 6422 raft_consensus.cc:359] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.905833 6422 raft_consensus.cc:385] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.906010 6422 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.906481 6422 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.906870 6544 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.908458 6422 ts_tablet_manager.cc:1434] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:56.909469 6422 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:56.912612 6544 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:56.914472 6421 tablet_bootstrap.cc:654] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.919559 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:56.920140 6229 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.920406 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:56.921160 6303 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:56.936492 6544 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.940552 6303 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:56.946388 6344 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:56.947664 6127 catalog_manager.cc:5696] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:56.949448 6544 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:56.951054 6544 raft_consensus.cc:697] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:56.952054 6527 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:56.951841 6544 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.974334 6422 tablet_bootstrap.cc:654] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:56.977138 6229 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
W20260620 19:57:56.981319 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:57:56.992390 6421 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:56.993019 6421 ts_tablet_manager.cc:1403] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.097s user 0.022s sys 0.038s
I20260620 19:57:56.997359 6421 raft_consensus.cc:359] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:56.998258 6421 raft_consensus.cc:385] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:56.998612 6421 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:56.999439 6421 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.002162 6423 tablet_bootstrap.cc:654] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.006538 6421 ts_tablet_manager.cc:1434] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260620 19:57:57.007476 6421 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.018728 6421 tablet_bootstrap.cc:654] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.020164 6523 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.020653 6523 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.020918 6127 catalog_manager.cc:5696] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:57.023931 6523 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.024444 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:57.025067 6381 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:57.026414 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.026998 6230 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:57.028139 6422 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.028721 6422 ts_tablet_manager.cc:1403] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.119s user 0.020s sys 0.010s
I20260620 19:57:57.027987 6273 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.030229 6429 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.030575 6429 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.030923 6429 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.037655 6427 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.043401 6429 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.038115 6427 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.047514 6427 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.048641 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.051826 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.052372 6229 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:57:57.053772 6345 leader_election.cc:343] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:57.055469 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.055962 6423 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.056388 6423 ts_tablet_manager.cc:1403] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.165s user 0.006s sys 0.009s
I20260620 19:57:57.057940 6429 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.058960 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:57:57.059186 6344 leader_election.cc:343] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:57.059496 6381 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.059638 6344 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:57.060698 6427 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:57.060606 6423 raft_consensus.cc:359] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.061300 6423 raft_consensus.cc:385] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.061556 6423 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.061825 6229 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:57.062883 6271 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:57.063087 6422 raft_consensus.cc:359] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.063697 6422 raft_consensus.cc:385] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.063966 6422 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.064604 6422 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.065718 6421 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:57.065502 6423 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.066124 6421 ts_tablet_manager.cc:1403] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.059s user 0.021s sys 0.010s
I20260620 19:57:57.066068 6381 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:57.066440 6523 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.066965 6523 raft_consensus.cc:697] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:57.067723 6523 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.068300 6423 ts_tablet_manager.cc:1434] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.012s user 0.005s sys 0.003s
I20260620 19:57:57.068713 6421 raft_consensus.cc:359] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.069424 6421 raft_consensus.cc:385] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.069679 6421 raft_consensus.cc:740] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.070592 6422 ts_tablet_manager.cc:1434] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20260620 19:57:57.070278 6421 consensus_queue.cc:260] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.069164 6423 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.072278 6421 ts_tablet_manager.cc:1434] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:57:57.073276 6421 tablet_bootstrap.cc:492] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.080776 6423 tablet_bootstrap.cc:654] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.081875 6421 tablet_bootstrap.cc:654] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.083272 6422 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.101424 6422 tablet_bootstrap.cc:654] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.100353 6128 catalog_manager.cc:5696] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:57.104605 6423 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.104938 6423 ts_tablet_manager.cc:1403] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.010s
I20260620 19:57:57.106796 6423 raft_consensus.cc:359] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.107409 6423 raft_consensus.cc:385] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.107669 6423 raft_consensus.cc:740] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.108285 6423 consensus_queue.cc:260] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.108744 6421 tablet_bootstrap.cc:492] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:57.109372 6421 ts_tablet_manager.cc:1403] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.015s
I20260620 19:57:57.110412 6423 ts_tablet_manager.cc:1434] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:57:57.111255 6423 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.112354 6421 raft_consensus.cc:359] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.113018 6421 raft_consensus.cc:385] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.113266 6421 raft_consensus.cc:740] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.114181 6421 consensus_queue.cc:260] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.116163 6421 ts_tablet_manager.cc:1434] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260620 19:57:57.116194 6423 tablet_bootstrap.cc:654] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.117239 6421 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.123616 6421 tablet_bootstrap.cc:654] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.124514 6422 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.124585 6523 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.125272 6422 ts_tablet_manager.cc:1403] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.000s
I20260620 19:57:57.125211 6523 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.126405 6429 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.126893 6429 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.134109 6427 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.134624 6523 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.134734 6427 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.136533 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.139092 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:57.140069 6381 raft_consensus.cc:2393] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:57.144960 6273 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:57.156025 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.156843 6229 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:57.157421 6523 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.158332 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:57.159417 6433 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.159668 6429 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.159922 6433 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.160421 6433 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.161015 6427 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.167021 6422 raft_consensus.cc:359] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.167901 6422 raft_consensus.cc:385] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.168229 6422 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.169185 6422 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.171031 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.171628 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:57.172133 6422 ts_tablet_manager.cc:1434] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20260620 19:57:57.172272 6381 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:57.173146 6422 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.171883 6229 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.164687 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.184484 6304 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.187342 6344 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.188293 6427 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.188670 6427 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.189018 6427 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.196967 6433 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.201273 6433 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.201731 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.202309 6229 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.203114 6421 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:57.203583 6422 tablet_bootstrap.cc:654] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.203568 6421 ts_tablet_manager.cc:1403] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.087s user 0.025s sys 0.028s
I20260620 19:57:57.207792 6523 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:57.208616 6493 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.209291 6427 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.212177 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.212859 6228 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.221390 6228 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.222860 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:57.226958 6544 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.227536 6544 raft_consensus.cc:697] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:57.228560 6544 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.242009 6229 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:57.247745 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:57.248389 6427 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.248513 6302 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.251849 6423 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.252316 6423 ts_tablet_manager.cc:1403] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.141s user 0.027s sys 0.048s
I20260620 19:57:57.253253 6493 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260620 19:57:57.254763 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:57.262208 6381 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.261435 6492 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260620 19:57:57.269711 6421 raft_consensus.cc:359] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.270412 6421 raft_consensus.cc:385] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.270700 6421 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.271373 6421 consensus_queue.cc:260] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.272123 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:57.272982 6523 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.273831 6523 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:57.274461 6492 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.274665 6523 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.282042 6128 catalog_manager.cc:5696] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:57.288542 6492 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.291429 6381 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:57.296896 6421 ts_tablet_manager.cc:1434] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.093s user 0.006s sys 0.000s
I20260620 19:57:57.298151 6421 tablet_bootstrap.cc:492] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.310238 6302 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.316109 6422 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.316584 6422 ts_tablet_manager.cc:1403] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.144s user 0.009s sys 0.028s
I20260620 19:57:57.322008 6422 raft_consensus.cc:359] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.323081 6422 raft_consensus.cc:385] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.323371 6422 raft_consensus.cc:740] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.323995 6423 raft_consensus.cc:359] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.324752 6423 raft_consensus.cc:385] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.324038 6422 consensus_queue.cc:260] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.325002 6423 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.325430 6129 catalog_manager.cc:5696] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:57.334365 6423 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.343436 6423 ts_tablet_manager.cc:1434] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.091s user 0.007s sys 0.001s
I20260620 19:57:57.344167 6423 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.344627 6421 tablet_bootstrap.cc:654] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.374799 6599 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.385902 6495 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.386407 6495 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.387849 6544 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.396047 6544 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.389281 6599 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.417421 6495 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.407349 6544 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.418644 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.419435 6302 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.420102 6544 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.417428 6564 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260620 19:57:57.423108 6599 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.423902 6344 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.425320 6427 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.425671 6427 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.426122 6427 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.430042 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.430747 6302 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.431936 6423 tablet_bootstrap.cc:654] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.432066 6344 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.434355 6544 raft_consensus.cc:2804] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.434806 6544 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.435146 6544 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.436372 6421 tablet_bootstrap.cc:492] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:57.436798 6421 ts_tablet_manager.cc:1403] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.139s user 0.028s sys 0.006s
I20260620 19:57:57.440117 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:57.441973 6422 ts_tablet_manager.cc:1434] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.125s user 0.008s sys 0.003s
I20260620 19:57:57.442296 6427 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.443462 6422 tablet_bootstrap.cc:492] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.447274 6529 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.448990 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:57.449548 6302 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.456791 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.457427 6228 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.458398 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.458931 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.459555 6229 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.459031 6230 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.460978 6427 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.465535 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.465932 6527 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.466183 6307 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:57.466364 6422 tablet_bootstrap.cc:654] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.466485 6495 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:57.469808 6544 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.471014 6376 raft_consensus.cc:2393] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:57.475107 6544 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.477108 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:57.477674 6306 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.480230 6195 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:57.482942 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.483577 6232 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.484141 6229 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.485606 6345 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:57.486681 6427 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.488049 6427 raft_consensus.cc:697] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:57.489075 6427 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.490540 6306 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.493646 6232 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.494105 6302 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.494722 6344 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.495177 6603 raft_consensus.cc:2764] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 19:57:57.495656 6527 raft_consensus.cc:2804] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.496222 6527 raft_consensus.cc:697] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:57.497119 6527 consensus_queue.cc:237] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.499735 6422 tablet_bootstrap.cc:492] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.500370 6422 ts_tablet_manager.cc:1403] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.002s
I20260620 19:57:57.505478 6421 raft_consensus.cc:359] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.510663 6421 raft_consensus.cc:385] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.510980 6421 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.511770 6421 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.518338 6421 ts_tablet_manager.cc:1434] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.081s user 0.009s sys 0.000s
I20260620 19:57:57.519659 6421 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.520684 6422 raft_consensus.cc:359] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.521394 6422 raft_consensus.cc:385] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.521659 6422 raft_consensus.cc:740] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.523953 6422 consensus_queue.cc:260] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.526451 6422 ts_tablet_manager.cc:1434] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260620 19:57:57.527441 6422 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.535100 6423 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.535637 6423 ts_tablet_manager.cc:1403] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.192s user 0.009s sys 0.028s
I20260620 19:57:57.535509 6126 catalog_manager.cc:5696] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:57.538143 6423 raft_consensus.cc:359] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.540977 6423 raft_consensus.cc:385] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.541252 6423 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.541919 6423 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.538918 6421 tablet_bootstrap.cc:654] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.538324 6126 catalog_manager.cc:5696] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:57.544198 6423 ts_tablet_manager.cc:1434] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260620 19:57:57.539863 6422 tablet_bootstrap.cc:654] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.545105 6423 tablet_bootstrap.cc:492] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.550702 6421 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:57.551200 6421 ts_tablet_manager.cc:1403] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.001s
I20260620 19:57:57.553714 6421 raft_consensus.cc:359] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.554531 6422 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.554476 6421 raft_consensus.cc:385] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.555066 6422 ts_tablet_manager.cc:1403] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.005s
I20260620 19:57:57.555182 6421 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.555966 6421 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.557413 6544 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.557633 6422 raft_consensus.cc:359] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.557866 6544 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.559967 6544 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.560530 6423 tablet_bootstrap.cc:654] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.561659 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.562692 6228 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.564436 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.565232 6345 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:57:57.566397 6344 leader_election.cc:343] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:57.566812 6544 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.567179 6544 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.567505 6544 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.571290 6423 tablet_bootstrap.cc:492] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.571664 6423 ts_tablet_manager.cc:1403] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.009s
I20260620 19:57:57.558298 6422 raft_consensus.cc:385] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.572414 6422 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.573055 6422 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.574097 6544 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.574254 6423 raft_consensus.cc:359] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.574862 6423 raft_consensus.cc:385] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.575115 6421 ts_tablet_manager.cc:1434] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260620 19:57:57.575109 6423 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.576128 6544 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.575912 6423 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.577054 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.577608 6228 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.578135 6423 ts_tablet_manager.cc:1434] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:57:57.579322 6422 ts_tablet_manager.cc:1434] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.024s user 0.004s sys 0.002s
I20260620 19:57:57.579560 6423 tablet_bootstrap.cc:492] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.580174 6422 tablet_bootstrap.cc:492] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.583319 6421 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.584460 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:57.587603 6228 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
W20260620 19:57:57.587970 6344 leader_election.cc:343] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:57.590741 6423 tablet_bootstrap.cc:654] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.593091 6345 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:57:57.594547 6544 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.595041 6544 raft_consensus.cc:697] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:57.595805 6544 consensus_queue.cc:237] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.598858 6421 tablet_bootstrap.cc:654] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.601289 6422 tablet_bootstrap.cc:654] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.603874 6423 tablet_bootstrap.cc:492] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.604300 6423 ts_tablet_manager.cc:1403] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.006s
I20260620 19:57:57.608145 6423 raft_consensus.cc:359] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.609014 6423 raft_consensus.cc:385] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.609335 6423 raft_consensus.cc:740] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.609815 6421 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:57.610246 6421 ts_tablet_manager.cc:1403] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.001s
I20260620 19:57:57.610248 6423 consensus_queue.cc:260] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.613981 6423 ts_tablet_manager.cc:1434] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20260620 19:57:57.615136 6423 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.622431 6423 tablet_bootstrap.cc:654] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.627410 6523 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.628618 6423 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.628494 6421 raft_consensus.cc:359] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.629050 6423 ts_tablet_manager.cc:1403] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260620 19:57:57.629169 6421 raft_consensus.cc:385] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.629429 6421 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.630539 6421 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.631313 6422 tablet_bootstrap.cc:492] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.631738 6422 ts_tablet_manager.cc:1403] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.020s
I20260620 19:57:57.632840 6421 ts_tablet_manager.cc:1434] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260620 19:57:57.632499 6423 raft_consensus.cc:359] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.633355 6423 raft_consensus.cc:385] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.633620 6423 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.633708 6421 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.634454 6423 consensus_queue.cc:260] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.634974 6422 raft_consensus.cc:359] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.635807 6422 raft_consensus.cc:385] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.636134 6422 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.636498 6423 ts_tablet_manager.cc:1434] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:57.621282 6130 catalog_manager.cc:5696] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:57.636854 6422 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.637409 6423 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.639060 6422 ts_tablet_manager.cc:1434] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:57:57.640090 6422 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.640219 6421 tablet_bootstrap.cc:654] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.647789 6423 tablet_bootstrap.cc:654] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.649394 6422 tablet_bootstrap.cc:654] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.660373 6523 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:57.686929 6429 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.687515 6429 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.689581 6429 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.690975 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:57.691648 6376 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:57.691833 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:57.693576 6271 leader_election.cc:343] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:57.697907 6423 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.698378 6423 ts_tablet_manager.cc:1403] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.061s user 0.031s sys 0.015s
I20260620 19:57:57.700277 6273 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:57.700914 6423 raft_consensus.cc:359] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.701584 6423 raft_consensus.cc:385] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.701691 6523 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.702232 6423 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.702508 6523 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.702883 6523 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.703186 6422 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.702939 6423 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.703646 6422 ts_tablet_manager.cc:1403] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.064s user 0.026s sys 0.027s
I20260620 19:57:57.705050 6423 ts_tablet_manager.cc:1434] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:57:57.705873 6423 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.706066 6422 raft_consensus.cc:359] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.706799 6422 raft_consensus.cc:385] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.707041 6422 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.707647 6422 consensus_queue.cc:260] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.709838 6422 ts_tablet_manager.cc:1434] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:57:57.714190 6529 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.714692 6529 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.716629 6529 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.716997 6523 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.718918 6433 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.720747 6421 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:57.722704 6421 ts_tablet_manager.cc:1403] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.008s
I20260620 19:57:57.720857 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.728868 6304 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.725612 6421 raft_consensus.cc:359] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.730142 6421 raft_consensus.cc:385] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.730473 6421 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.731175 6421 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.732554 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.732820 6421 ts_tablet_manager.cc:1434] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260620 19:57:57.733153 6228 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.733887 6421 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:57.722108 6523 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.733744 6344 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.721422 6422 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.734966 6529 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.735726 6529 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.736166 6529 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.740716 6228 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:57.741808 6271 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:57.742543 6523 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.743041 6523 raft_consensus.cc:697] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:57.743893 6423 tablet_bootstrap.cc:654] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.744743 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:57.745333 6374 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.743921 6523 consensus_queue.cc:237] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.754771 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.755600 6232 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.759130 6374 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:57.762487 6422 tablet_bootstrap.cc:654] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.765151 6529 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.770732 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.771504 6232 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.783192 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:57.783840 6302 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.787647 6427 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.788116 6427 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.788745 6544 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.790040 6427 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.786649 6130 catalog_manager.cc:5696] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:57.792740 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.793991 6304 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.793469 6529 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.795142 6344 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.794170 6632 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260620 19:57:57.800087 6427 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.801083 6427 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.805114 6427 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.806432 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.807061 6230 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:57.807766 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:57:57.809772 6344 leader_election.cc:343] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:57.805883 6529 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:57.813144 6529 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.813484 6529 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.830245 6423 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:57.830801 6423 ts_tablet_manager.cc:1403] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.125s user 0.032s sys 0.030s
I20260620 19:57:57.833369 6544 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.836148 6302 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.837180 6421 tablet_bootstrap.cc:654] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.838876 6422 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:57.839711 6422 ts_tablet_manager.cc:1403] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.118s user 0.016s sys 0.005s
I20260620 19:57:57.837682 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.841135 6344 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:57:57.846138 6345 leader_election.cc:343] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:57.841905 6632 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.846769 6345 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:57.852092 6632 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.852731 6527 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:57.853521 6544 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:57.852022 6232 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.856952 6544 raft_consensus.cc:697] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:57.859277 6544 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.859552 6423 raft_consensus.cc:359] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.860145 6423 raft_consensus.cc:385] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.860390 6423 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.861019 6423 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.863552 6423 ts_tablet_manager.cc:1434] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.033s user 0.003s sys 0.009s
I20260620 19:57:57.864380 6423 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:57.870666 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.871537 6232 raft_consensus.cc:2393] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:57.857167 6632 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.873041 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:57.875417 6422 raft_consensus.cc:359] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.876072 6422 raft_consensus.cc:385] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:57.876327 6422 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:57.881778 6273 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:57.882573 6632 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.891054 6422 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.891997 6632 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:57.897360 6423 tablet_bootstrap.cc:654] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:57.882220 6529 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.900857 6422 ts_tablet_manager.cc:1434] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.061s user 0.004s sys 0.001s
I20260620 19:57:57.901746 6422 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:57.903386 6529 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.903652 6527 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.904119 6527 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.906216 6527 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:57.908509 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:57.909533 6302 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.910049 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:57.910656 6304 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.915688 6344 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:57.916136 6427 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:57.916649 6527 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:57:57.918218 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:57.917155 6527 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:57.918817 6527 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:57.923827 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:57.924693 6230 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.918789 6228 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:57.939615 6632 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.940119 6632 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.941998 6130 catalog_manager.cc:5696] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:57.955991 6228 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:57.965776 6607 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.966482 6607 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.977561 6523 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:57.978199 6523 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:57.980391 6523 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:57.981856 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.038280 6345 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.054651 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.068830 6381 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:58.069129 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:58.069322 6495 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:58.069770 6304 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:58.069840 6495 raft_consensus.cc:697] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:58.009768 6523 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:57:58.067490 6271 leader_election.cc:343] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.034616 6492 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:57:58.039106 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.021780 6422 tablet_bootstrap.cc:654] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.058934 6423 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:58.039278 6421 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:58.070765 6495 consensus_queue.cc:237] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.039613 6607 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.085799 6421 ts_tablet_manager.cc:1403] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.351s user 0.011s sys 0.029s
I20260620 19:57:58.089022 6421 raft_consensus.cc:359] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.089748 6421 raft_consensus.cc:385] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.090086 6421 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.090544 6529 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.090809 6421 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.093148 6421 ts_tablet_manager.cc:1434] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 19:57:58.093216 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:57:58.093422 6273 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:58.094506 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.095140 6381 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:58.036875 6302 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:58.095535 6429 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:58.095997 6429 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.065061 6527 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.096307 6273 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:58.096413 6429 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.097072 6433 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:57:58.097440 6433 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.097780 6433 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:58.098063 6527 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:58.050675 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.100059 6230 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:58.084645 6523 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.094115 6421 tablet_bootstrap.cc:492] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:58.085055 6423 ts_tablet_manager.cc:1403] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.221s user 0.015s sys 0.005s
I20260620 19:57:58.048926 6632 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.105115 6376 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:58.111737 6433 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.112277 6198 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.113231 6607 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:57:58.113574 6607 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.113905 6607 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:58.115116 6429 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.084913 6492 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.115718 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.118729 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.119284 6381 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:58.120914 6607 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.117326 6433 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.122918 6230 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 2: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:58.126798 6607 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.135273 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.135423 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.135694 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:58.131212 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:58.140931 6228 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:58.140990 6306 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:58.141404 6527 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.136683 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:58.143510 6302 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:58.137352 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.130988 6427 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260620 19:57:58.143953 6128 catalog_manager.cc:5696] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:58.139719 6495 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.140478 6303 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:57:58.138346 6423 raft_consensus.cc:359] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.147825 6423 raft_consensus.cc:385] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.148186 6423 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.148988 6423 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.132627 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.155803 6232 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:58.156539 6381 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:57:58.138782 6492 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.165781 6527 consensus_queue.cc:1048] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:58.167768 6271 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.169332 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.169333 6273 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:58.171262 6228 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:57:58.173686 6195 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.152669 6523 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.187695 6433 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:57:58.188076 6523 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:58.188107 6433 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.188627 6523 raft_consensus.cc:697] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:58.182693 6345 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.183694 6306 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:57:58.184216 6429 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.184690 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.185652 6643 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:57:58.187615 6421 tablet_bootstrap.cc:654] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.188700 6433 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:58.179704 6423 ts_tablet_manager.cc:1434] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.076s user 0.006s sys 0.001s
I20260620 19:57:58.193456 6643 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:57:58.194558 6643 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:57:58.195313 6198 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:58.157725 6495 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.193514 6427 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:57:58.204449 6632 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.204869 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.215250 6423 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:58.212993 6492 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:57:58.216579 6381 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.223306 6374 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:58.216966 6427 raft_consensus.cc:697] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:58.224915 6427 consensus_queue.cc:237] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.226082 6495 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:58.227823 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.220649 6523 consensus_queue.cc:237] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.216442 6632 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.237579 6632 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.254297 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.254978 6378 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
W20260620 19:57:58.247066 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 65834.2us
W20260620 19:57:58.264494 6271 leader_election.cc:343] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.265775 6523 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.266224 6523 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.267876 6273 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:58.268219 6523 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.268999 6429 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:58.269366 6429 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.269737 6429 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.238906 6433 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.272862 6433 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.273581 6643 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.274941 6232 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:57:58.276335 6643 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.279700 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.281611 6422 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:58.275656 6128 catalog_manager.cc:5696] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:58.282413 6422 ts_tablet_manager.cc:1403] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.381s user 0.011s sys 0.021s
I20260620 19:57:58.285395 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:58.286190 6304 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:58.290709 6427 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:57:58.293192 6645 consensus_queue.cc:1048] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.296574 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.297284 6374 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:58.298098 6423 tablet_bootstrap.cc:654] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.295480 6427 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.302563 6427 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:58.297911 6429 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.304059 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.304522 6422 raft_consensus.cc:359] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.305595 6422 raft_consensus.cc:385] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.305857 6422 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.306492 6422 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.311786 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.312459 6232 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:58.314343 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:58.315116 6302 raft_consensus.cc:2368] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
W20260620 19:57:58.317930 6650 log.cc:927] Time spent T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260620 19:57:58.318148 6195 leader_election.cc:400] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:57:58.318490 6195 leader_election.cc:403] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260620 19:57:58.319216 6422 ts_tablet_manager.cc:1434] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20260620 19:57:58.319419 6607 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:57:58.300611 6381 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:58.318308 6273 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:58.321075 6422 tablet_bootstrap.cc:492] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:58.321770 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.323688 6421 tablet_bootstrap.cc:492] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:58.324324 6421 ts_tablet_manager.cc:1403] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.230s user 0.022s sys 0.025s
I20260620 19:57:58.323171 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.327821 6421 raft_consensus.cc:359] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.328562 6421 raft_consensus.cc:385] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.328826 6421 raft_consensus.cc:740] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.329568 6421 consensus_queue.cc:260] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.331732 6421 ts_tablet_manager.cc:1434] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:58.331853 6632 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:58.332401 6421 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:58.332350 6632 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.332886 6632 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.324424 6429 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.334472 6427 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.340016 6607 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:57:58.340102 6374 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:58.341545 6427 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:58.342568 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:58.343333 6302 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:58.344574 6632 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.349411 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.350423 6427 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.351801 6427 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.352496 6523 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.351962 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:58.355410 6645 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:57:58.351233 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.357059 6232 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.357141 6273 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:58.358419 6433 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:58.358991 6433 raft_consensus.cc:697] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:58.359925 6433 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.351236 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.364773 6374 raft_consensus.cc:2368] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:57:58.399829 6228 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:57:58.412408 6429 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:58.411680 6427 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:58.420313 6422 tablet_bootstrap.cc:654] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.425447 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:57:58.427834 6271 leader_election.cc:343] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.426038 6374 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.433450 6632 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.438221 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:58.444734 6345 leader_election.cc:343] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.446208 6427 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.446744 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.447331 6376 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.449460 6423 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:58.449975 6423 ts_tablet_manager.cc:1403] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.235s user 0.012s sys 0.010s
I20260620 19:57:58.450034 6374 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:58.455930 6421 tablet_bootstrap.cc:654] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.461071 6376 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:58.462430 6232 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:58.472470 6273 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:58.473459 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.472581 6198 leader_election.cc:400] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:57:58.453336 6423 raft_consensus.cc:359] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.482234 6423 raft_consensus.cc:385] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.482522 6423 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
W20260620 19:57:58.482769 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 54030us
I20260620 19:57:58.483295 6423 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.486169 6645 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.486034 6130 catalog_manager.cc:5696] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 2, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:58.488099 6433 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:58.488749 6433 raft_consensus.cc:697] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:58.489935 6423 ts_tablet_manager.cc:1434] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260620 19:57:58.489563 6433 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.491181 6423 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:58.494796 6271 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:58.486008 6422 tablet_bootstrap.cc:492] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:58.497699 6422 ts_tablet_manager.cc:1403] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.177s user 0.010s sys 0.009s
I20260620 19:57:58.499008 6632 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:57:58.499496 6632 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:58.499622 6130 catalog_manager.cc:5696] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:58.500289 6632 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.506873 6643 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.507870 6643 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.509816 6643 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:57:58.510862 6271 leader_election.cc:343] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.511584 6423 tablet_bootstrap.cc:654] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.511716 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:57:58.516580 6198 leader_election.cc:343] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.519047 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:58.523053 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:57:58.523602 6195 leader_election.cc:343] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.524021 6195 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:58.524909 6492 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:58.500301 6422 raft_consensus.cc:359] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.527128 6422 raft_consensus.cc:385] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.527560 6273 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:58.528671 6422 raft_consensus.cc:740] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.530476 6422 consensus_queue.cc:260] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.533866 6422 ts_tablet_manager.cc:1434] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20260620 19:57:58.534766 6422 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:58.541452 6423 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:58.541904 6423 ts_tablet_manager.cc:1403] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.011s
I20260620 19:57:58.544273 6422 tablet_bootstrap.cc:654] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.550872 6654 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:58.551363 6654 raft_consensus.cc:697] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:58.553072 6421 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:58.552603 6654 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.553514 6421 ts_tablet_manager.cc:1403] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.221s user 0.017s sys 0.005s
I20260620 19:57:58.556071 6421 raft_consensus.cc:359] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.557096 6421 raft_consensus.cc:385] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.557452 6421 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.558935 6421 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:57:58.560889 6344 leader_election.cc:343] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.561378 6344 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:58.562376 6645 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:58.562794 6423 raft_consensus.cc:359] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.563097 6544 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.563449 6423 raft_consensus.cc:385] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.563728 6423 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.564211 6529 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.564332 6423 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.565908 6423 ts_tablet_manager.cc:1434] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20260620 19:57:58.566766 6423 tablet_bootstrap.cc:492] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:58.587002 6421 ts_tablet_manager.cc:1434] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20260620 19:57:58.588027 6421 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:58.589725 6423 tablet_bootstrap.cc:654] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.598474 6130 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:58.600407 6130 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:58.602453 6422 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:58.602073 6130 catalog_manager.cc:5696] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:58.603623 6421 tablet_bootstrap.cc:654] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.605331 6422 ts_tablet_manager.cc:1403] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.071s user 0.020s sys 0.018s
I20260620 19:57:58.612097 6422 raft_consensus.cc:359] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.612838 6422 raft_consensus.cc:385] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.613112 6422 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.613729 6422 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.616160 6422 ts_tablet_manager.cc:1434] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260620 19:57:58.618049 6422 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:58.624125 6654 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.624713 6654 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.628952 6654 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.630940 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:57:58.632665 6273 leader_election.cc:343] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.633239 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:58.634783 6271 leader_election.cc:343] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.635201 6271 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:58.636296 6433 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:58.641736 6422 tablet_bootstrap.cc:654] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.645272 6421 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:58.648025 6421 ts_tablet_manager.cc:1403] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.007s
I20260620 19:57:58.651997 6421 raft_consensus.cc:359] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.652787 6232 raft_consensus.cc:1275] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260620 19:57:58.653004 6421 raft_consensus.cc:385] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.654012 6421 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.654495 6658 consensus_queue.cc:1048] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:58.655170 6421 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.658003 6421 ts_tablet_manager.cc:1434] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260620 19:57:58.659327 6423 tablet_bootstrap.cc:492] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:58.661098 6423 ts_tablet_manager.cc:1403] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.008s
I20260620 19:57:58.660513 6421 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:58.669598 6421 tablet_bootstrap.cc:654] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.674635 6423 raft_consensus.cc:359] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.675089 6433 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.675374 6423 raft_consensus.cc:385] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.675714 6423 raft_consensus.cc:740] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.675578 6433 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.676441 6423 consensus_queue.cc:260] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.677618 6433 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.678591 6423 ts_tablet_manager.cc:1434] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20260620 19:57:58.678897 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.679641 6423 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:58.679793 6643 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.680657 6643 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.681934 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.685230 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.697580 6376 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
W20260620 19:57:58.687824 6271 leader_election.cc:343] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.689556 6422 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:58.698860 6198 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.699476 6422 ts_tablet_manager.cc:1403] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.082s user 0.016s sys 0.012s
W20260620 19:57:58.699834 6273 leader_election.cc:343] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.682585 6643 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.700515 6643 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:58.700867 6643 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.701187 6643 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.702013 6422 raft_consensus.cc:359] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.702143 6273 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:58.691766 6423 tablet_bootstrap.cc:654] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.692432 6381 raft_consensus.cc:1275] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260620 19:57:58.703495 6658 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:58.695403 6421 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:58.704284 6422 raft_consensus.cc:385] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.709395 6422 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.710018 6422 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.708947 6421 ts_tablet_manager.cc:1403] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.009s
I20260620 19:57:58.711308 6306 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260620 19:57:58.711498 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:58.712074 6302 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:58.712960 6422 ts_tablet_manager.cc:1434] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260620 19:57:58.712839 6544 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.713863 6422 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:58.715718 6658 consensus_queue.cc:1048] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.717068 6421 raft_consensus.cc:359] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.717816 6421 raft_consensus.cc:385] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.718120 6421 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.720820 6421 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.726966 6421 ts_tablet_manager.cc:1434] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20260620 19:57:58.728298 6421 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:58.734679 6421 tablet_bootstrap.cc:654] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.734364 6643 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.743379 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:58.743959 6306 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.745255 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.745836 6376 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.749694 6643 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.751536 6423 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:58.752002 6423 ts_tablet_manager.cc:1403] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.073s user 0.021s sys 0.013s
I20260620 19:57:58.752260 6306 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:58.752369 6376 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:58.753131 6421 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:58.753594 6421 ts_tablet_manager.cc:1403] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.006s
I20260620 19:57:58.753712 6198 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:58.755084 6643 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:58.754848 6423 raft_consensus.cc:359] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.759306 6423 raft_consensus.cc:385] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.759596 6423 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.760501 6422 tablet_bootstrap.cc:654] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.761268 6643 raft_consensus.cc:697] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:58.762032 6421 raft_consensus.cc:359] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.762176 6643 consensus_queue.cc:237] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.762807 6421 raft_consensus.cc:385] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.763199 6421 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.763852 6421 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.766283 6421 ts_tablet_manager.cc:1434] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20260620 19:57:58.760178 6423 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.767309 6421 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:58.774466 6232 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260620 19:57:58.781471 6421 tablet_bootstrap.cc:654] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.784751 6495 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260620 19:57:58.789117 6232 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.792212 6423 ts_tablet_manager.cc:1434] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.040s user 0.006s sys 0.002s
I20260620 19:57:58.793185 6423 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:58.798018 6422 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:58.798483 6422 ts_tablet_manager.cc:1403] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.085s user 0.017s sys 0.022s
I20260620 19:57:58.800956 6422 raft_consensus.cc:359] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.801587 6422 raft_consensus.cc:385] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.801865 6422 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.805317 6422 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.816927 6433 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.819243 6423 tablet_bootstrap.cc:654] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.820595 6126 catalog_manager.cc:5696] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:57:58.824105 6422 ts_tablet_manager.cc:1434] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.025s user 0.004s sys 0.001s
I20260620 19:57:58.825173 6422 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:58.832434 6422 tablet_bootstrap.cc:654] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.845578 6433 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.877104 6423 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:58.877583 6423 ts_tablet_manager.cc:1403] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.020s
I20260620 19:57:58.880201 6423 raft_consensus.cc:359] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.880995 6423 raft_consensus.cc:385] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.881255 6423 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.882313 6421 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:58.882771 6421 ts_tablet_manager.cc:1403] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.116s user 0.011s sys 0.015s
I20260620 19:57:58.882752 6423 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.884848 6423 ts_tablet_manager.cc:1434] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 19:57:58.885351 6421 raft_consensus.cc:359] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.885874 6423 tablet_bootstrap.cc:492] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:58.886011 6421 raft_consensus.cc:385] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.886332 6421 raft_consensus.cc:740] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.886976 6421 consensus_queue.cc:260] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.889096 6421 ts_tablet_manager.cc:1434] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:57:58.890872 6421 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:58.891976 6423 tablet_bootstrap.cc:654] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.894282 6643 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.894762 6643 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.896852 6643 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.897887 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:58.898370 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:58.898998 6306 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
W20260620 19:57:58.900810 6198 leader_election.cc:343] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.904564 6195 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:58.905961 6607 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:58.906314 6607 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.906757 6607 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.907339 6228 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:57:58.910488 6433 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.911896 6376 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:57:58.913314 6433 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:57:58.913959 6658 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.922655 6607 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.926802 6495 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:58.927322 6495 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.929704 6495 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:58.930743 6423 tablet_bootstrap.cc:492] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:58.931186 6423 ts_tablet_manager.cc:1403] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.010s
I20260620 19:57:58.931046 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:58.931622 6306 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:58.933574 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:58.934032 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:58.934032 6423 raft_consensus.cc:359] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.934723 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:58.935029 6423 raft_consensus.cc:385] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.935350 6607 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:58.935456 6423 raft_consensus.cc:740] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:58.933585 6344 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:58.936044 6422 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:58.936512 6422 ts_tablet_manager.cc:1403] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.112s user 0.023s sys 0.017s
I20260620 19:57:58.936765 6527 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:58.936394 6423 consensus_queue.cc:260] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.937191 6527 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:58.937613 6527 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.934836 6306 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.938988 6423 ts_tablet_manager.cc:1434] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260620 19:57:58.938977 6345 leader_election.cc:343] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.940181 6423 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:58.942409 6422 raft_consensus.cc:359] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.943084 6422 raft_consensus.cc:385] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:58.943315 6422 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
W20260620 19:57:58.946933 6198 leader_election.cc:343] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:58.952193 6422 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.956823 6421 tablet_bootstrap.cc:654] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.958540 6422 ts_tablet_manager.cc:1434] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260620 19:57:58.959455 6422 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:58.975875 6527 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:58.977720 6658 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:58.980291 6423 tablet_bootstrap.cc:654] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:58.981935 6527 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:58.982749 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:58.983685 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:58.990212 6304 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:58.998792 6230 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.013345 6306 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:57:59.011572 6658 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.010932 6495 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.025288 6195 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:59.026757 6429 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.028661 6422 tablet_bootstrap.cc:654] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.033318 6230 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:59.034171 6658 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.040014 6304 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:59.043382 6345 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:59.041072 6495 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.044247 6527 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.044860 6527 raft_consensus.cc:697] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:59.045817 6495 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.045899 6527 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.047730 6607 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.051611 6607 raft_consensus.cc:697] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:57:59.052807 6607 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.053215 6658 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.062463 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.063344 6306 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:59.064373 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.064966 6232 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:59.066299 6271 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:59.067529 6433 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.067889 6433 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.068222 6433 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.070312 6344 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.071821 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.072408 6376 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:59.073238 6495 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.073246 6421 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.073930 6495 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.078109 6495 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.076007 6421 ts_tablet_manager.cc:1403] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.185s user 0.017s sys 0.041s
I20260620 19:57:59.077806 6423 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.082149 6423 ts_tablet_manager.cc:1403] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.142s user 0.019s sys 0.037s
I20260620 19:57:59.058986 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.082496 6421 raft_consensus.cc:359] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.083014 6230 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:59.083251 6421 raft_consensus.cc:385] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.083662 6421 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.084615 6427 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.084328 6421 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.087040 6427 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.087182 6423 raft_consensus.cc:359] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.087936 6423 raft_consensus.cc:385] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.087918 6421 ts_tablet_manager.cc:1434] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260620 19:57:59.088337 6423 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.088881 6495 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.089339 6421 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.089905 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.096102 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:59.096658 6228 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.097728 6421 tablet_bootstrap.cc:654] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.103478 6228 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:59.103421 6643 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.104780 6433 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.105280 6643 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.107224 6643 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.107406 6529 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.107815 6529 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.108662 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.109584 6232 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.110574 6529 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.111042 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:59.111678 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.111837 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.113710 6195 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:59.090054 6427 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.114967 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.103099 6128 catalog_manager.cc:5696] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
W20260620 19:57:59.104553 6345 leader_election.cc:343] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:57:59.116240 6344 leader_election.cc:343] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.116978 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:59.118685 6345 leader_election.cc:343] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.119151 6345 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.120409 6645 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.123718 6345 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:59.124593 6427 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.125249 6427 raft_consensus.cc:697] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:59.125653 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:59.127360 6273 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.128166 6429 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.126276 6427 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.129724 6495 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.131788 6607 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.132382 6423 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.137178 6423 ts_tablet_manager.cc:1434] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.055s user 0.002s sys 0.003s
I20260620 19:57:59.137034 6304 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:59.140818 6423 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.141521 6495 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.141997 6495 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.143373 6529 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.144604 6529 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.145509 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:59.146225 6304 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:59.147606 6429 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:59.149371 6344 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.150056 6645 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.150403 6645 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.150734 6645 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.153280 6495 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.154695 6529 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.156165 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:59.157378 6345 leader_election.cc:343] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.161383 6645 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.163182 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:59.179272 6345 leader_election.cc:343] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.180197 6423 tablet_bootstrap.cc:654] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.180912 6433 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.185616 6645 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.187125 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:59.187878 6421 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.188337 6421 ts_tablet_manager.cc:1403] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.099s user 0.028s sys 0.047s
I20260620 19:57:59.189946 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.190568 6304 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
W20260620 19:57:59.190819 6345 leader_election.cc:343] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.192863 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.195288 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:59.195891 6306 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.200141 6344 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:57:59.201659 6344 leader_election.cc:343] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.205694 6344 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.204124 6427 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.208791 6427 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.209216 6427 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.209745 6527 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.212893 6128 catalog_manager.cc:5696] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:59.239347 6306 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:59.241294 6492 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:59.241715 6344 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.253301 6544 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.253795 6544 raft_consensus.cc:697] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:59.254570 6544 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.260093 6421 raft_consensus.cc:359] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.260592 6421 raft_consensus.cc:385] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.260836 6421 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.258564 6427 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.261937 6421 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.268658 6422 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:59.271378 6130 catalog_manager.cc:5696] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:59.278955 6422 ts_tablet_manager.cc:1403] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.320s user 0.010s sys 0.020s
I20260620 19:57:59.277396 6421 ts_tablet_manager.cc:1434] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.089s user 0.004s sys 0.004s
I20260620 19:57:59.280776 6421 tablet_bootstrap.cc:492] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.280846 6427 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.281944 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:59.282531 6303 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.279733 6642 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.285533 6642 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.287153 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:57:59.292989 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:57:59.296561 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:57:59.298008 6273 leader_election.cc:343] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.293416 6642 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.306269 6423 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.306815 6423 ts_tablet_manager.cc:1403] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.166s user 0.021s sys 0.043s
I20260620 19:57:59.311524 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.314925 6303 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:59.314883 6423 raft_consensus.cc:359] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.315747 6423 raft_consensus.cc:385] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.316023 6423 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.316653 6423 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.319048 6423 ts_tablet_manager.cc:1434] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260620 19:57:59.319741 6423 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.322257 6422 raft_consensus.cc:359] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.324235 6422 raft_consensus.cc:385] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.325858 6422 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.326694 6344 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.329375 6422 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:57:59.329375 6271 leader_election.cc:343] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.331050 6271 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.332190 6228 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.329540 6529 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.335924 6529 raft_consensus.cc:697] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:59.337549 6529 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.338191 6422 ts_tablet_manager.cc:1434] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20260620 19:57:59.339176 6422 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:59.332552 6642 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.349793 6643 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:59.352572 6228 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:59.360077 6421 tablet_bootstrap.cc:654] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.365744 6423 tablet_bootstrap.cc:654] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.372051 6422 tablet_bootstrap.cc:654] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.388592 6129 catalog_manager.cc:5696] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:59.394275 6129 catalog_manager.cc:5696] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:59.406891 6421 tablet_bootstrap.cc:492] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.407361 6421 ts_tablet_manager.cc:1403] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.127s user 0.023s sys 0.013s
I20260620 19:57:59.409467 6421 raft_consensus.cc:359] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.409947 6421 raft_consensus.cc:385] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.410209 6421 raft_consensus.cc:740] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.410957 6421 consensus_queue.cc:260] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.414077 6421 ts_tablet_manager.cc:1434] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 19:57:59.414875 6421 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.417887 6422 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:59.418066 6423 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.418329 6422 ts_tablet_manager.cc:1403] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.079s user 0.015s sys 0.003s
I20260620 19:57:59.418737 6423 ts_tablet_manager.cc:1403] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.099s user 0.014s sys 0.015s
I20260620 19:57:59.421617 6423 raft_consensus.cc:359] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.422039 6421 tablet_bootstrap.cc:654] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.421933 6422 raft_consensus.cc:359] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.422446 6423 raft_consensus.cc:385] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.422672 6422 raft_consensus.cc:385] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.422994 6422 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.423820 6422 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.422899 6423 raft_consensus.cc:740] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.428694 6421 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.428543 6423 consensus_queue.cc:260] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.429191 6421 ts_tablet_manager.cc:1403] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260620 19:57:59.430866 6423 ts_tablet_manager.cc:1434] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260620 19:57:59.431753 6423 tablet_bootstrap.cc:492] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.431854 6421 raft_consensus.cc:359] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.432495 6421 raft_consensus.cc:385] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.432750 6421 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.433465 6421 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.435120 6422 ts_tablet_manager.cc:1434] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260620 19:57:59.435570 6421 ts_tablet_manager.cc:1434] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:57:59.436751 6421 tablet_bootstrap.cc:492] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.440366 6422 tablet_bootstrap.cc:492] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:59.442878 6529 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.443354 6529 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.446991 6529 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.447139 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:59.448526 6345 leader_election.cc:343] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.450642 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.451275 6302 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:57:59.452258 6344 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.453192 6529 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.453553 6529 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.453884 6529 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.462536 6423 tablet_bootstrap.cc:654] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.465320 6422 tablet_bootstrap.cc:654] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.468577 6529 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.471171 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:57:59.473001 6345 leader_election.cc:343] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.476656 6421 tablet_bootstrap.cc:654] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.479153 6529 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.480229 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:59.481483 6303 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.482234 6423 tablet_bootstrap.cc:492] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.482889 6423 ts_tablet_manager.cc:1403] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.014s
I20260620 19:57:59.486441 6423 raft_consensus.cc:359] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.488415 6423 raft_consensus.cc:385] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.488726 6423 raft_consensus.cc:740] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.489580 6422 tablet_bootstrap.cc:492] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:59.489486 6423 consensus_queue.cc:260] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.490000 6422 ts_tablet_manager.cc:1403] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.019s
I20260620 19:57:59.492038 6423 ts_tablet_manager.cc:1434] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260620 19:57:59.490585 6303 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:57:59.492581 6422 raft_consensus.cc:359] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.493204 6422 raft_consensus.cc:385] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.493474 6422 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.493815 6344 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.493865 6423 tablet_bootstrap.cc:492] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.494688 6422 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.495661 6529 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.496292 6529 raft_consensus.cc:697] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:57:59.497465 6422 ts_tablet_manager.cc:1434] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260620 19:57:59.497311 6529 consensus_queue.cc:237] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.498819 6422 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:59.514075 6423 tablet_bootstrap.cc:654] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.514436 6422 tablet_bootstrap.cc:654] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.516888 6421 tablet_bootstrap.cc:492] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.517378 6421 ts_tablet_manager.cc:1403] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.081s user 0.020s sys 0.022s
I20260620 19:57:59.538872 6422 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:59.539309 6529 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.540028 6422 ts_tablet_manager.cc:1403] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.014s
I20260620 19:57:59.541296 6423 tablet_bootstrap.cc:492] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.544519 6423 ts_tablet_manager.cc:1403] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.051s user 0.017s sys 0.009s
I20260620 19:57:59.545291 6422 raft_consensus.cc:359] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.546038 6422 raft_consensus.cc:385] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.546316 6422 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.541441 6421 raft_consensus.cc:359] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.546962 6421 raft_consensus.cc:385] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.547241 6421 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.547443 6423 raft_consensus.cc:359] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.548269 6423 raft_consensus.cc:385] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.548569 6423 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.549227 6423 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.549381 6421 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.551326 6423 ts_tablet_manager.cc:1434] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260620 19:57:59.551564 6422 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.552385 6423 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.553701 6422 ts_tablet_manager.cc:1434] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20260620 19:57:59.554616 6422 tablet_bootstrap.cc:492] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:59.551326 6421 ts_tablet_manager.cc:1434] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.034s user 0.005s sys 0.003s
I20260620 19:57:59.558717 6423 tablet_bootstrap.cc:654] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.559659 6421 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.558640 6128 catalog_manager.cc:5696] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:57:59.566582 6421 tablet_bootstrap.cc:654] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.570886 6422 tablet_bootstrap.cc:654] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.588115 6422 tablet_bootstrap.cc:492] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:59.588115 6423 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.588644 6422 ts_tablet_manager.cc:1403] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.001s
I20260620 19:57:59.588783 6423 ts_tablet_manager.cc:1403] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.003s
I20260620 19:57:59.591747 6423 raft_consensus.cc:359] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.592505 6423 raft_consensus.cc:385] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.592855 6423 raft_consensus.cc:740] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.593518 6423 consensus_queue.cc:260] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.594060 6421 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.594496 6421 ts_tablet_manager.cc:1403] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.004s
I20260620 19:57:59.595860 6423 ts_tablet_manager.cc:1434] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:57:59.596912 6421 raft_consensus.cc:359] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.597409 6423 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.597579 6421 raft_consensus.cc:385] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.597899 6421 raft_consensus.cc:740] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.598518 6421 consensus_queue.cc:260] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.599614 6427 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.602149 6421 ts_tablet_manager.cc:1434] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260620 19:57:59.603027 6421 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.603514 6423 tablet_bootstrap.cc:654] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.591928 6422 raft_consensus.cc:359] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.605170 6422 raft_consensus.cc:385] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.605469 6422 raft_consensus.cc:740] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.606124 6422 consensus_queue.cc:260] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.609057 6422 ts_tablet_manager.cc:1434] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.020s user 0.005s sys 0.004s
I20260620 19:57:59.610098 6422 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:59.611758 6643 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.617050 6423 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.617487 6423 ts_tablet_manager.cc:1403] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20260620 19:57:59.620429 6423 raft_consensus.cc:359] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.620868 6422 tablet_bootstrap.cc:654] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.621083 6423 raft_consensus.cc:385] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.621354 6423 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.622045 6423 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.623123 6421 tablet_bootstrap.cc:654] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.624099 6423 ts_tablet_manager.cc:1434] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:57:59.626293 6423 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.626931 6529 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:57:59.631224 6632 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.631942 6632 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.633533 6423 tablet_bootstrap.cc:654] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.637054 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:59.639001 6271 leader_election.cc:343] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.639070 6422 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:59.640822 6422 ts_tablet_manager.cc:1403] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.000s
I20260620 19:57:59.642741 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.643329 6381 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:59.644049 6422 raft_consensus.cc:359] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.644778 6422 raft_consensus.cc:385] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.644630 6273 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.645213 6422 raft_consensus.cc:740] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.645848 6642 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.645884 6422 consensus_queue.cc:260] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.646343 6642 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.646637 6423 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.647097 6423 ts_tablet_manager.cc:1403] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.014s
I20260620 19:57:59.646901 6642 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.648651 6421 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.649057 6421 ts_tablet_manager.cc:1403] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.000s
I20260620 19:57:59.649703 6423 raft_consensus.cc:359] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.650339 6423 raft_consensus.cc:385] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.650633 6423 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.651479 6421 raft_consensus.cc:359] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.651449 6423 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.652138 6421 raft_consensus.cc:385] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.652427 6421 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.653158 6421 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.654254 6423 ts_tablet_manager.cc:1434] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:57:59.655144 6423 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.655144 6421 ts_tablet_manager.cc:1434] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260620 19:57:59.656365 6421 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.661121 6423 tablet_bootstrap.cc:654] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.663892 6422 ts_tablet_manager.cc:1434] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260620 19:57:59.665081 6422 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:57:59.668740 6632 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.671775 6422 tablet_bootstrap.cc:654] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.672312 6381 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.673257 6423 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.673784 6423 ts_tablet_manager.cc:1403] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260620 19:57:59.671797 6421 tablet_bootstrap.cc:654] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.676352 6423 raft_consensus.cc:359] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.677037 6423 raft_consensus.cc:385] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.677295 6423 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.678040 6423 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.678588 6544 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.680662 6423 ts_tablet_manager.cc:1434] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:57:59.681744 6423 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:57:59.685340 6643 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.685819 6643 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.688265 6643 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.689569 6423 tablet_bootstrap.cc:654] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:57:59.689989 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.690129 6544 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.690670 6376 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:57:59.691672 6198 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:57:59.692718 6643 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.693493 6643 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.693673 6632 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.693818 6643 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.694810 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.701159 6632 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.707490 6658 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.707931 6658 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.709646 6658 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.710875 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.711185 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.711467 6376 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
W20260620 19:57:59.712628 6273 leader_election.cc:343] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.713126 6273 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.713697 6658 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.714020 6658 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.714341 6658 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.714763 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.715880 6529 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.721062 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.727388 6529 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.730392 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:57:59.731623 6345 leader_election.cc:343] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.732580 6433 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.733165 6433 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.735263 6433 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.737567 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.738206 6376 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:59.739833 6273 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.740548 6433 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.740861 6433 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.741534 6433 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.743008 6527 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.732579 6529 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.749640 6433 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.754195 6433 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.751844 6658 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.760980 6645 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.761473 6645 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.763321 6658 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.755662 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:59.763499 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:59.764688 6376 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.764712 6378 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.769783 6607 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.770370 6607 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.771518 6376 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:59.772869 6607 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.773083 6273 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.774722 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.775290 6429 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.775719 6429 raft_consensus.cc:697] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:59.776448 6429 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.778417 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.780681 6645 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:57:59.784629 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.785481 6303 raft_consensus.cc:2393] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:57:59.784135 6433 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.787599 6433 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.789366 6433 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.793313 6195 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:59.794353 6607 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.794626 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.795291 6377 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:57:59.797214 6378 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:59.797417 6273 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.799222 6433 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:57:59.799614 6433 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:57:59.799968 6433 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.803414 6607 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.803479 6273 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.804510 6658 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.805423 6658 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:59.806406 6658 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.814532 6433 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.816203 6433 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.818786 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:59.821717 6544 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:57:59.821816 6429 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.822809 6429 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.824822 6429 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.826776 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:57:59.829150 6273 leader_election.cc:343] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.830437 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:59.831130 6378 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.843603 6642 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.844729 6306 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:59.754817 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.755578 6632 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.857117 6492 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.857560 6492 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.859887 6492 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.861840 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.862496 6306 raft_consensus.cc:2393] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:59.863310 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.865041 6195 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:57:59.865417 6643 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.866060 6492 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.867419 6643 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.871750 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:57:59.872568 6271 leader_election.cc:343] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.872865 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:57:59.873409 6271 leader_election.cc:343] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.873797 6271 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.882141 6607 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.882588 6607 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.884433 6607 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.889510 6378 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:57:59.897823 6642 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.898427 6658 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 19:57:59.899639 6271 leader_election.cc:343] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:57:59.900382 6271 leader_election.cc:343] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.910069 6422 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:57:59.910542 6422 ts_tablet_manager.cc:1403] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.246s user 0.027s sys 0.073s
I20260620 19:57:59.913762 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:59.914207 6421 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:57:59.914841 6421 ts_tablet_manager.cc:1403] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.259s user 0.044s sys 0.055s
I20260620 19:57:59.914902 6643 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.915452 6643 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.917563 6643 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:57:59.918465 6421 raft_consensus.cc:359] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.919363 6421 raft_consensus.cc:385] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.919713 6421 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.919659 6422 raft_consensus.cc:359] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.964764 6423 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:57:59.925330 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.971495 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.969107 6423 ts_tablet_manager.cc:1403] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.288s user 0.089s sys 0.120s
I20260620 19:57:59.957676 6227 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:57:59.944069 6273 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:57:59.920754 6421 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.969586 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:59.940016 6492 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 19:57:59.942756 6344 leader_election.cc:343] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.974030 6344 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.976110 6344 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:57:59.957993 6544 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.922381 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:57:59.960024 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:59.968659 6422 raft_consensus.cc:385] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:57:59.976858 6422 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.955212 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:57:59.979143 6226 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:57:59.924264 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:57:59.980412 6421 ts_tablet_manager.cc:1434] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.065s user 0.013s sys 0.002s
I20260620 19:57:59.982386 6422 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.983351 6433 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:57:59.981390 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:57:59.983835 6433 raft_consensus.cc:697] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:57:59.984002 6306 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:57:59.954722 6733 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20260620 19:57:59.984576 6433 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:57:59.980846 6527 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.990449 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:57:59.998687 6421 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:57:59.983474 6495 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:00.010573 6423 raft_consensus.cc:359] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.011657 6423 raft_consensus.cc:385] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.012072 6423 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:57:59.983377 6376 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.003870 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.005192 6198 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.007647 6422 ts_tablet_manager.cc:1434] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.097s user 0.006s sys 0.000s
I20260620 19:57:59.989292 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:57:59.986410 6230 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:00.014457 6423 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.028182 6422 tablet_bootstrap.cc:492] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
W20260620 19:58:00.035279 6198 leader_election.cc:343] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:57:59.997689 6658 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:57:59.981968 6544 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.034803 6733 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:57:59.988039 6529 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20260620 19:58:00.045951 6423 ts_tablet_manager.cc:1434] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.074s user 0.005s sys 0.002s
I20260620 19:58:00.046984 6423 tablet_bootstrap.cc:492] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Bootstrap starting.
W20260620 19:58:00.048012 6195 leader_election.cc:343] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.048437 6195 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:58:00.050719 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 60519.5us
W20260620 19:58:00.051854 6271 leader_election.cc:343] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
W20260620 19:58:00.055416 6271 leader_election.cc:343] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.055858 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.056416 6429 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.051184 6301 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:00.040021 6658 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.049871 6733 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.091131 6741 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.097927 6421 tablet_bootstrap.cc:654] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.097971 6195 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.114452 6658 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.115933 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.122095 6423 tablet_bootstrap.cc:654] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.116492 6607 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.099167 6741 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.125771 6739 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.127780 6739 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.129289 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.130658 6374 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:00.128448 6529 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260620 19:58:00.131495 6544 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:00.131824 6739 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:00.128981 6271 leader_election.cc:343] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.133313 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.138417 6422 tablet_bootstrap.cc:654] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.141144 6273 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:00.145596 6345 leader_election.cc:343] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.147513 6423 tablet_bootstrap.cc:492] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:00.147954 6423 ts_tablet_manager.cc:1403] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.101s user 0.013s sys 0.009s
I20260620 19:58:00.143582 6642 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.149997 6741 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.148336 6642 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.150686 6642 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.162364 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.162942 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:00.165728 6271 leader_election.cc:343] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.167363 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.165246 6642 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:00.169085 6273 leader_election.cc:343] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.169484 6273 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.170397 6658 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.159801 6376 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:00.174808 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:00.177191 6642 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.178900 6273 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:00.178465 6345 leader_election.cc:343] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.181185 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:00.181867 6378 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.190330 6740 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:00.190874 6740 raft_consensus.cc:697] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:58:00.191431 6271 leader_election.cc:343] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.199100 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.200595 6302 raft_consensus.cc:2393] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:00.202540 6126 catalog_manager.cc:5696] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.204281 6126 catalog_manager.cc:5696] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.205446 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.206051 6303 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:58:00.191644 6740 consensus_queue.cc:237] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.205942 6126 catalog_manager.cc:5696] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.209052 6378 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:00.214790 6423 raft_consensus.cc:359] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.215656 6423 raft_consensus.cc:385] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.215925 6423 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.216600 6423 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.217552 6344 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.221676 6344 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.222419 6529 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.222927 6529 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.223356 6529 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.224279 6645 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.230157 6422 tablet_bootstrap.cc:492] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:00.232457 6421 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:00.244516 6421 ts_tablet_manager.cc:1403] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.252s user 0.016s sys 0.015s
I20260620 19:58:00.247184 6529 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.247565 6273 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.249224 6529 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:00.249887 6422 ts_tablet_manager.cc:1403] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.225s user 0.018s sys 0.010s
I20260620 19:58:00.249236 6632 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:00.278311 6423 ts_tablet_manager.cc:1434] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.130s user 0.007s sys 0.000s
I20260620 19:58:00.280123 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:00.278323 6495 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260620 19:58:00.285054 6632 raft_consensus.cc:697] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:58:00.285322 6345 leader_election.cc:343] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.281168 6422 raft_consensus.cc:359] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.287365 6422 raft_consensus.cc:385] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.288196 6422 raft_consensus.cc:740] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.283030 6423 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:00.286374 6632 consensus_queue.cc:237] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.289958 6740 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.290473 6740 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.295192 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:00.295998 6302 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:58:00.300680 6271 consensus_peers.cc:597] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 -> Peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Couldn't send request to peer eb2f228ebe40440d877be6302bba2cd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:00.301203 6745 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.305091 6421 raft_consensus.cc:359] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.305776 6421 raft_consensus.cc:385] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.306043 6421 raft_consensus.cc:740] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.301726 6745 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.306558 6421 consensus_queue.cc:260] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.308046 6422 consensus_queue.cc:260] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.309866 6421 ts_tablet_manager.cc:1434] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.065s user 0.004s sys 0.000s
I20260620 19:58:00.310742 6421 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:00.319097 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.319759 6227 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:00.324839 6271 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:00.327299 6422 ts_tablet_manager.cc:1434] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.077s user 0.010s sys 0.010s
I20260620 19:58:00.328018 6422 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:00.328809 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.329480 6376 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:00.336162 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.337502 6745 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.318440 6740 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.338295 6632 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.338677 6632 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.339937 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.341821 6376 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:58:00.342199 6302 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:00.339013 6632 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.344748 6423 tablet_bootstrap.cc:654] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.345191 6198 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:00.346812 6344 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.347836 6495 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:00.349967 6747 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.350158 6495 raft_consensus.cc:697] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:00.351087 6495 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.350368 6747 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.352500 6747 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.353026 6741 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:00.374913 6743 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.375519 6743 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.379721 6743 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:00.383347 6271 consensus_peers.cc:597] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 -> Peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Couldn't send request to peer eb2f228ebe40440d877be6302bba2cd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:00.384806 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.385344 6306 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:58:00.385710 6741 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:58:00.387894 6195 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:00.391824 6607 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.392215 6607 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.392555 6607 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.390082 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.392370 6632 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:00.393834 6198 leader_election.cc:343] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.397866 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:00.399535 6422 tablet_bootstrap.cc:654] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.402956 6658 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:00.404071 6305 raft_consensus.cc:2393] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:00.415216 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:00.415807 6381 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.424520 6607 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.422319 6381 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:00.428357 6273 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:00.415786 6632 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.429773 6741 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:00.430266 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:00.430965 6741 raft_consensus.cc:697] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:58:00.431623 6198 leader_election.cc:343] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.432008 6741 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.428465 6607 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.432880 6421 tablet_bootstrap.cc:654] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.432449 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:00.436005 6305 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.442225 6740 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.442874 6740 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.445626 6747 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.450212 6226 raft_consensus.cc:2393] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:00.439276 6632 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260620 19:58:00.465497 6423 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:00.465936 6423 ts_tablet_manager.cc:1403] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.183s user 0.012s sys 0.006s
I20260620 19:58:00.473224 6747 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.476310 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:00.477087 6376 raft_consensus.cc:2393] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:00.478480 6740 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.481845 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:00.483346 6273 leader_election.cc:343] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.487447 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.488020 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:00.488052 6226 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:00.490160 6195 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.491250 6747 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.495230 6271 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.500749 6423 raft_consensus.cc:359] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.501598 6423 raft_consensus.cc:385] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.501889 6423 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.502460 6632 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.504297 6632 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.505446 6632 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.514266 6422 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:00.514773 6422 ts_tablet_manager.cc:1403] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.187s user 0.020s sys 0.010s
I20260620 19:58:00.518644 6421 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:00.519089 6421 ts_tablet_manager.cc:1403] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.209s user 0.015s sys 0.014s
I20260620 19:58:00.512362 6632 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.525269 6421 raft_consensus.cc:359] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.525931 6421 raft_consensus.cc:385] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.526302 6421 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.526924 6421 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.527275 6632 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.529019 6421 ts_tablet_manager.cc:1434] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260620 19:58:00.529878 6421 tablet_bootstrap.cc:492] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:00.533274 6423 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.532212 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:00.536911 6423 ts_tablet_manager.cc:1434] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.071s user 0.003s sys 0.004s
I20260620 19:58:00.537724 6422 raft_consensus.cc:359] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.538511 6422 raft_consensus.cc:385] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.538838 6422 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.548568 6423 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:00.548337 6226 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.552198 6129 catalog_manager.cc:5696] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.553922 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:00.553958 6129 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.557135 6129 catalog_manager.cc:5696] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:00.566387 6273 leader_election.cc:343] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.569522 6421 tablet_bootstrap.cc:654] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.569886 6305 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:00.571053 6226 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:00.574880 6271 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.575917 6632 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:00.576381 6632 raft_consensus.cc:697] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:00.577157 6632 consensus_queue.cc:237] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.581482 6423 tablet_bootstrap.cc:654] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.581372 6195 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.582270 6643 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:00.583098 6643 raft_consensus.cc:697] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:00.583869 6643 consensus_queue.cc:237] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.589577 6130 catalog_manager.cc:5696] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:00.590408 6422 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.592736 6422 ts_tablet_manager.cc:1434] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.077s user 0.010s sys 0.005s
I20260620 19:58:00.593642 6422 tablet_bootstrap.cc:492] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:00.601982 6422 tablet_bootstrap.cc:654] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.613924 6130 catalog_manager.cc:5696] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.615790 6632 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.616292 6632 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.629940 6129 catalog_manager.cc:5696] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.635388 6421 tablet_bootstrap.cc:492] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:00.635857 6421 ts_tablet_manager.cc:1403] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.106s user 0.009s sys 0.004s
I20260620 19:58:00.636168 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:00.638797 6271 leader_election.cc:343] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.638969 6765 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260620 19:58:00.639768 6632 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.642362 6423 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:00.642643 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.642830 6423 ts_tablet_manager.cc:1403] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.001s
I20260620 19:58:00.643244 6381 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:00.645466 6423 raft_consensus.cc:359] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.646134 6423 raft_consensus.cc:385] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.646529 6423 raft_consensus.cc:740] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.648761 6423 consensus_queue.cc:260] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.651270 6423 ts_tablet_manager.cc:1434] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 19:58:00.651440 6422 tablet_bootstrap.cc:492] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:00.651971 6422 ts_tablet_manager.cc:1403] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.002s
I20260620 19:58:00.652554 6423 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:00.657096 6273 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.657821 6765 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.658190 6765 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.658540 6765 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.661013 6422 raft_consensus.cc:359] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.661732 6422 raft_consensus.cc:385] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.662011 6422 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.665943 6422 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.666177 6765 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.667578 6422 ts_tablet_manager.cc:1434] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.015s user 0.004s sys 0.001s
I20260620 19:58:00.668462 6422 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:00.670274 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:00.671849 6271 leader_election.cc:343] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.672943 6747 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.673429 6747 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.675467 6747 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.677892 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.659725 6421 raft_consensus.cc:359] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:00.685891 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:58:00.687150 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:00.687776 6423 tablet_bootstrap.cc:654] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.687592 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.687857 6381 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.679862 6632 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.688773 6632 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.690805 6422 tablet_bootstrap.cc:654] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.691478 6632 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.679047 6765 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.692726 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.692800 6195 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.694046 6747 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.695353 6645 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.695835 6645 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:00.696954 6271 leader_election.cc:343] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.698920 6765 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:00.700461 6645 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:00.700825 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.701447 6378 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:00.702946 6273 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.703511 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.704147 6300 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:58:00.704715 6632 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.705066 6632 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.705410 6632 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.706851 6421 raft_consensus.cc:385] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.707209 6421 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.707895 6421 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.709398 6421 ts_tablet_manager.cc:1434] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.073s user 0.002s sys 0.003s
I20260620 19:58:00.710126 6421 tablet_bootstrap.cc:492] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:00.710474 6344 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:00.712222 6381 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:00.714186 6529 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.714542 6529 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.714655 6273 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.714913 6529 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.715664 6765 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:00.716382 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.716732 6765 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:00.717621 6765 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:00.719053 6345 leader_election.cc:343] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.722540 6529 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.727032 6529 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:00.727754 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:00.726647 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:00.727895 6632 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.728804 6421 tablet_bootstrap.cc:654] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.729266 6305 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:58:00.729993 6345 leader_election.cc:343] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.732796 6344 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.734683 6529 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.735699 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:00.743974 6271 leader_election.cc:343] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.746033 6632 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.747839 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:00.748564 6381 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:00.749006 6422 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:00.749773 6273 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.749819 6422 ts_tablet_manager.cc:1403] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.005s
I20260620 19:58:00.753866 6632 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.757073 6423 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:00.757493 6423 ts_tablet_manager.cc:1403] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.105s user 0.023s sys 0.005s
I20260620 19:58:00.757493 6421 tablet_bootstrap.cc:492] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:00.758157 6421 ts_tablet_manager.cc:1403] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.000s
I20260620 19:58:00.760248 6421 raft_consensus.cc:359] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.760124 6423 raft_consensus.cc:359] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.760965 6421 raft_consensus.cc:385] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.761261 6421 raft_consensus.cc:740] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.761093 6423 raft_consensus.cc:385] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.761585 6423 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.761889 6421 consensus_queue.cc:260] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.761832 6126 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:00.762230 6423 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.764411 6423 ts_tablet_manager.cc:1434] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:58:00.766206 6423 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:00.766881 6643 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.767318 6643 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.768049 6421 ts_tablet_manager.cc:1434] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260620 19:58:00.769367 6643 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.769887 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.772492 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.774821 6422 raft_consensus.cc:359] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.775544 6422 raft_consensus.cc:385] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.775857 6422 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.776620 6195 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.776669 6422 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.777900 6643 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.779059 6422 ts_tablet_manager.cc:1434] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260620 19:58:00.779778 6421 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:00.779969 6422 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:00.782275 6423 tablet_bootstrap.cc:654] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.791115 6422 tablet_bootstrap.cc:654] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.791740 6421 tablet_bootstrap.cc:654] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.795639 6423 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:00.796052 6423 ts_tablet_manager.cc:1403] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.006s
I20260620 19:58:00.798457 6423 raft_consensus.cc:359] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.799096 6423 raft_consensus.cc:385] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.799358 6423 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.799988 6423 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.802007 6423 ts_tablet_manager.cc:1434] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:58:00.802939 6423 tablet_bootstrap.cc:492] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:00.803458 6226 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.806285 6422 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:00.806846 6422 ts_tablet_manager.cc:1403] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.003s
I20260620 19:58:00.808877 6423 tablet_bootstrap.cc:654] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.809773 6422 raft_consensus.cc:359] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.810849 6422 raft_consensus.cc:385] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.811167 6422 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.811949 6422 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.814261 6422 ts_tablet_manager.cc:1434] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260620 19:58:00.815177 6422 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:00.821664 6422 tablet_bootstrap.cc:654] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.830111 6421 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:00.830530 6421 ts_tablet_manager.cc:1403] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.009s
I20260620 19:58:00.830724 6423 tablet_bootstrap.cc:492] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:00.831156 6423 ts_tablet_manager.cc:1403] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.004s
I20260620 19:58:00.832953 6421 raft_consensus.cc:359] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.833621 6421 raft_consensus.cc:385] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.834000 6421 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.833659 6423 raft_consensus.cc:359] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.834373 6423 raft_consensus.cc:385] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.834678 6423 raft_consensus.cc:740] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.834775 6421 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.841117 6421 ts_tablet_manager.cc:1434] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260620 19:58:00.842023 6421 tablet_bootstrap.cc:492] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:00.842326 6423 consensus_queue.cc:260] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.845284 6423 ts_tablet_manager.cc:1434] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260620 19:58:00.846208 6423 tablet_bootstrap.cc:492] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:00.847633 6421 tablet_bootstrap.cc:654] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.852576 6423 tablet_bootstrap.cc:654] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.854842 6429 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:00.861215 6529 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:00.875829 6226 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.880427 6750 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.880941 6750 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.884761 6750 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.885991 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.886776 6376 raft_consensus.cc:2393] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:00.888345 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:00.890681 6198 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.892095 6643 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.895489 6747 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.895973 6747 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.899109 6747 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.899364 6643 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:00.902446 6422 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:00.902689 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:00.902954 6271 consensus_peers.cc:597] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 -> Peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Couldn't send request to peer eb2f228ebe40440d877be6302bba2cd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:00.905004 6422 ts_tablet_manager.cc:1403] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.019s
I20260620 19:58:00.905808 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:00.907549 6645 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:00.907858 6198 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:00.908963 6422 raft_consensus.cc:359] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.909255 6743 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:00.910084 6422 raft_consensus.cc:385] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.910423 6422 raft_consensus.cc:740] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.911144 6422 consensus_queue.cc:260] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.918418 6421 tablet_bootstrap.cc:492] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:00.918874 6421 ts_tablet_manager.cc:1403] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.000s
I20260620 19:58:00.921273 6421 raft_consensus.cc:359] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.921779 6421 raft_consensus.cc:385] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.922034 6421 raft_consensus.cc:740] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.923142 6765 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:00.923746 6765 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.925704 6423 tablet_bootstrap.cc:492] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:00.926996 6423 ts_tablet_manager.cc:1403] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.013s
I20260620 19:58:00.928418 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:00.926563 6765 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:00.931108 6423 raft_consensus.cc:359] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.930347 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:00.931057 6271 leader_election.cc:343] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:00.931819 6423 raft_consensus.cc:385] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:00.932090 6423 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:00.932657 6381 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:00.932696 6423 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.926574 6421 consensus_queue.cc:260] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.960204 6423 ts_tablet_manager.cc:1434] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.033s user 0.004s sys 0.000s
I20260620 19:58:00.960335 6421 ts_tablet_manager.cc:1434] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20260620 19:58:00.960927 6423 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:00.961016 6421 tablet_bootstrap.cc:492] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:00.938628 6741 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:00.939455 6273 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:00.943423 6422 ts_tablet_manager.cc:1434] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20260620 19:58:00.964535 6422 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:00.969520 6765 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:00.969949 6765 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:00.970274 6765 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.977944 6421 tablet_bootstrap.cc:654] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.980561 6422 tablet_bootstrap.cc:654] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.983608 6423 tablet_bootstrap.cc:654] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:00.984404 6765 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:00.997115 6228 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:00.991019 6741 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260620 19:58:00.998472 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:00.999042 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:01.001739 6765 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.002666 6378 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:58:01.002682 6271 leader_election.cc:343] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.026191 6421 tablet_bootstrap.cc:492] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:01.026714 6421 ts_tablet_manager.cc:1403] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.002s
I20260620 19:58:01.034238 6381 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.036749 6422 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:01.037230 6422 ts_tablet_manager.cc:1403] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.011s
I20260620 19:58:01.045288 6422 raft_consensus.cc:359] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.045950 6422 raft_consensus.cc:385] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.046974 6421 raft_consensus.cc:359] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.047508 6421 raft_consensus.cc:385] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.047765 6421 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.048307 6421 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.024911 6378 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:01.046236 6422 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.050449 6422 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.053314 6422 ts_tablet_manager.cc:1434] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260620 19:58:01.053678 6741 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:01.055359 6422 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:01.059866 6421 ts_tablet_manager.cc:1434] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260620 19:58:01.060763 6421 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:01.072636 6273 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.077189 6422 tablet_bootstrap.cc:654] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.078027 6741 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:01.078866 6741 raft_consensus.cc:697] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:01.079648 6741 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.084545 6423 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:01.084981 6423 ts_tablet_manager.cc:1403] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.124s user 0.001s sys 0.017s
I20260620 19:58:01.087337 6750 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.087806 6750 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.089766 6750 leader_election.cc:290] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.087810 6423 raft_consensus.cc:359] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.092535 6423 raft_consensus.cc:385] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.092811 6423 raft_consensus.cc:740] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.093575 6423 consensus_queue.cc:260] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.094911 6377 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.095683 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:01.096500 6376 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:01.097988 6658 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.098620 6658 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.099673 6423 ts_tablet_manager.cc:1434] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260620 19:58:01.101161 6658 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.102169 6421 tablet_bootstrap.cc:654] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.107944 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:01.109967 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:01.110821 6226 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:01.112279 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:01.113018 6300 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:01.113461 6740 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:58:01.117267 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:01.119932 6195 leader_election.cc:304] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:01.122886 6765 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:01.123298 6765 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:01.122917 6747 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:58:01.123641 6765 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:58:01.121611 6273 leader_election.cc:343] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.124547 6433 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.125146 6433 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.124608 6750 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.138754 6433 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.149499 6495 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.159749 6529 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:01.189714 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 67225.3us
I20260620 19:58:01.193117 6529 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.194294 6423 tablet_bootstrap.cc:492] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:01.194202 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:01.192637 6495 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.196607 6422 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:01.197055 6422 ts_tablet_manager.cc:1403] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.142s user 0.006s sys 0.005s
I20260620 19:58:01.197664 6495 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:01.198201 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:01.199424 6345 leader_election.cc:343] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.201367 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:01.201399 6271 leader_election.cc:343] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.201912 6306 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:58:01.202756 6344 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.203459 6495 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:01.203827 6495 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:01.204178 6495 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.215354 6529 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:01.215929 6495 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.218158 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:01.219764 6344 leader_election.cc:343] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.221714 6765 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.222261 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:01.222836 6306 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.223534 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:01.223757 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:01.224747 6345 leader_election.cc:343] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.225109 6345 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.225970 6544 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 19:58:01.226843 6345 leader_election.cc:343] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.229175 6306 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:01.230129 6495 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:01.230923 6344 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.231455 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:01.233146 6544 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:01.233548 6544 raft_consensus.cc:697] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:01.234247 6544 consensus_queue.cc:237] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.235160 6422 raft_consensus.cc:359] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.235761 6422 raft_consensus.cc:385] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.236024 6422 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.236225 6750 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:01.236635 6422 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.240458 6765 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.242380 6422 ts_tablet_manager.cc:1434] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.045s user 0.004s sys 0.003s
I20260620 19:58:01.243875 6422 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
W20260620 19:58:01.245637 6273 leader_election.cc:343] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.246062 6273 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.245796 6128 catalog_manager.cc:5696] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:01.252303 6741 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:01.253808 6747 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260620 19:58:01.260741 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:01.267335 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:01.260934 6765 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260620 19:58:01.268572 6232 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.283134 6232 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:01.287379 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:01.289131 6433 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 19:58:01.289285 6273 leader_election.cc:343] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.289631 6433 raft_consensus.cc:697] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:01.292692 6423 tablet_bootstrap.cc:654] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.290383 6433 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.278229 6740 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260620 19:58:01.288936 6658 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.300818 6658 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.313174 6421 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:01.313802 6421 ts_tablet_manager.cc:1403] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.253s user 0.011s sys 0.010s
I20260620 19:58:01.318022 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:01.319518 6271 leader_election.cc:343] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.321743 6126 catalog_manager.cc:5696] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:01.325625 6658 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:01.331786 6800 log.cc:927] Time spent T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260620 19:58:01.340801 6421 raft_consensus.cc:359] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.341414 6421 raft_consensus.cc:385] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.341648 6421 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.340375 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:01.343709 6273 leader_election.cc:343] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.343890 6422 tablet_bootstrap.cc:654] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.344120 6273 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.344400 6421 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.366273 6421 ts_tablet_manager.cc:1434] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.052s user 0.010s sys 0.013s
I20260620 19:58:01.367183 6421 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:01.367383 6128 catalog_manager.cc:5696] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:01.368291 6740 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:01.376351 6423 tablet_bootstrap.cc:492] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:01.376927 6423 ts_tablet_manager.cc:1403] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.183s user 0.018s sys 0.013s
I20260620 19:58:01.380208 6421 tablet_bootstrap.cc:654] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.387467 6423 raft_consensus.cc:359] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.388180 6423 raft_consensus.cc:385] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.388533 6423 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.389338 6423 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.399591 6423 ts_tablet_manager.cc:1434] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20260620 19:58:01.400676 6423 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: Bootstrap starting.
I20260620 19:58:01.400636 6740 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.401227 6740 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.403235 6740 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.403898 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:01.405354 6271 leader_election.cc:343] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.405581 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:01.406365 6377 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 0.
I20260620 19:58:01.407632 6273 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.408476 6740 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:01.408842 6740 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:01.409189 6740 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.411684 6423 tablet_bootstrap.cc:654] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.415482 6421 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:01.415902 6421 ts_tablet_manager.cc:1403] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.006s
I20260620 19:58:01.418382 6421 raft_consensus.cc:359] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.419049 6421 raft_consensus.cc:385] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.419363 6421 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.419983 6421 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.422115 6421 ts_tablet_manager.cc:1434] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260620 19:58:01.423041 6421 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:01.423381 6740 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.426621 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:01.427167 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:01.427932 6377 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:58:01.428334 6271 leader_election.cc:343] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.429389 6421 tablet_bootstrap.cc:654] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.438915 6377 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:01.440896 6273 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.441900 6741 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:01.442513 6741 raft_consensus.cc:697] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:01.443583 6741 consensus_queue.cc:237] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.449836 6421 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:01.450297 6421 ts_tablet_manager.cc:1403] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.018s
I20260620 19:58:01.452747 6421 raft_consensus.cc:359] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.453395 6421 raft_consensus.cc:385] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.453688 6421 raft_consensus.cc:740] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.460291 6422 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:01.460763 6422 ts_tablet_manager.cc:1403] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.217s user 0.001s sys 0.023s
I20260620 19:58:01.463579 6422 raft_consensus.cc:359] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.464224 6422 raft_consensus.cc:385] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.464480 6422 raft_consensus.cc:740] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.465147 6422 consensus_queue.cc:260] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.464403 6421 consensus_queue.cc:260] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.471095 6740 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.472317 6422 ts_tablet_manager.cc:1434] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260620 19:58:01.473065 6421 ts_tablet_manager.cc:1434] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260620 19:58:01.473963 6421 tablet_bootstrap.cc:492] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:01.478974 6422 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:01.484194 6421 tablet_bootstrap.cc:654] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.517036 6643 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.517694 6643 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.519634 6423 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: No bootstrap required, opened a new log
I20260620 19:58:01.519838 6422 tablet_bootstrap.cc:654] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.520695 6423 ts_tablet_manager.cc:1403] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: Time spent bootstrapping tablet: real 0.120s user 0.025s sys 0.031s
I20260620 19:58:01.520481 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:01.522010 6305 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:58:01.521920 6130 catalog_manager.cc:5696] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:01.523353 6643 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.523753 6195 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:01.524317 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:01.524602 6643 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:01.526784 6643 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:01.527354 6643 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.528206 6421 tablet_bootstrap.cc:492] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:01.528630 6421 ts_tablet_manager.cc:1403] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.013s
I20260620 19:58:01.525555 6377 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 0.
I20260620 19:58:01.541546 6421 raft_consensus.cc:359] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.542199 6421 raft_consensus.cc:385] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.542475 6421 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.543107 6421 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.551609 6421 ts_tablet_manager.cc:1434] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20260620 19:58:01.552627 6421 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Bootstrap starting.
I20260620 19:58:01.555123 6643 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.557886 6643 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.560470 6421 tablet_bootstrap.cc:654] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.560534 6422 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:01.561038 6422 ts_tablet_manager.cc:1403] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.082s user 0.010s sys 0.001s
I20260620 19:58:01.561183 6423 raft_consensus.cc:359] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.561887 6423 raft_consensus.cc:385] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.562065 6423 raft_consensus.cc:740] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.563871 6423 consensus_queue.cc:260] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.564936 6422 raft_consensus.cc:359] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.565866 6422 raft_consensus.cc:385] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.566205 6422 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.567025 6422 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.568202 6423 ts_tablet_manager.cc:1434] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: Time spent starting tablet: real 0.047s user 0.005s sys 0.002s
I20260620 19:58:01.570011 6422 ts_tablet_manager.cc:1434] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260620 19:58:01.641289 6422 tablet_bootstrap.cc:492] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:01.643605 6529 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.646296 6529 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.649736 6529 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:01.645113 6421 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: No bootstrap required, opened a new log
I20260620 19:58:01.650769 6544 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.651104 6421 ts_tablet_manager.cc:1403] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent bootstrapping tablet: real 0.099s user 0.020s sys 0.000s
I20260620 19:58:01.651743 6544 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.654772 6544 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:01.657047 6643 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.656939 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:01.660400 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:01.660558 6643 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:01.662099 6345 leader_election.cc:343] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.664870 6643 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.666566 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:01.666874 6345 leader_election.cc:343] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.667685 6305 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:58:01.677224 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:01.677683 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:01.678664 6306 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:58:01.679241 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:01.680079 6377 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.679610 6421 raft_consensus.cc:359] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.680310 6344 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.681427 6529 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:01.681753 6529 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:01.682060 6529 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.682411 6422 tablet_bootstrap.cc:654] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.684599 6344 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:01.680518 6421 raft_consensus.cc:385] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.691185 6421 raft_consensus.cc:740] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.691805 6544 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:01.692217 6544 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:01.692541 6544 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.693729 6421 consensus_queue.cc:260] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.707352 6421 ts_tablet_manager.cc:1434] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent starting tablet: real 0.056s user 0.010s sys 0.006s
I20260620 19:58:01.718206 6375 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.729452 6377 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:01.731868 6529 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.732911 6198 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:01.734160 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:01.734953 6306 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.741925 6544 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.744228 6544 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:01.744916 6747 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:01.745899 6747 raft_consensus.cc:697] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:01.751683 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:01.756778 6422 tablet_bootstrap.cc:492] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:01.757360 6422 ts_tablet_manager.cc:1403] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.117s user 0.013s sys 0.005s
I20260620 19:58:01.759002 6529 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:01.760361 6306 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:01.762936 6740 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:58:01.765480 6747 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.776964 6422 raft_consensus.cc:359] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.777957 6422 raft_consensus.cc:385] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:01.778299 6422 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:01.779145 6422 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.783519 6422 ts_tablet_manager.cc:1434] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.026s user 0.006s sys 0.005s
I20260620 19:58:01.785739 6422 tablet_bootstrap.cc:492] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:01.799713 6195 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:01.804009 6750 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:01.809665 6740 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:58:01.811650 6747 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:01.817955 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:01.818615 6230 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.829320 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:01.812253 6747 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.831244 6227 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.832513 6747 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:01.835338 6229 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.837723 6230 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:01.848918 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:01.850339 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:01.851059 6300 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.858867 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:01.859440 6306 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:01.868175 6300 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:01.872278 6422 tablet_bootstrap.cc:654] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:01.876049 6645 consensus_queue.cc:1048] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260620 19:58:01.876997 6178 auto_rebalancer.cc:259] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260620 19:58:01.879194 6344 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:01.880784 6529 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:01.881291 6529 raft_consensus.cc:697] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:01.882222 6529 consensus_queue.cc:237] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.883857 6740 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:01.887254 6306 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
W20260620 19:58:01.888070 6345 leader_election.cc:343] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:01.888932 6345 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:01.890690 6544 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:01.891237 6544 raft_consensus.cc:697] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:01.892136 6544 consensus_queue.cc:237] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:01.904755 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:01.924778 6345 consensus_peers.cc:597] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Couldn't send request to peer eb2f228ebe40440d877be6302bba2cd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:01.942584 6740 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:01.946496 6195 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:01.956022 6747 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 19:58:02.000288 6271 consensus_peers.cc:597] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 -> Peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Couldn't send request to peer eb2f228ebe40440d877be6302bba2cd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260620 19:58:02.007266 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 128068us
I20260620 19:58:02.102046 6422 tablet_bootstrap.cc:492] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:02.102675 6422 ts_tablet_manager.cc:1403] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.317s user 0.016s sys 0.014s
I20260620 19:58:02.109166 6422 raft_consensus.cc:359] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.120877 6422 raft_consensus.cc:385] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:02.121169 6422 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:02.129415 6422 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.130059 6429 consensus_queue.cc:1048] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260620 19:58:02.165475 6825 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:02.174885 6825 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.176995 6825 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:02.177503 6128 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:02.182334 6422 ts_tablet_manager.cc:1434] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.079s user 0.008s sys 0.000s
I20260620 19:58:02.183423 6422 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:02.194869 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:02.195387 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:02.195631 6299 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 0.
I20260620 19:58:02.199055 6344 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:02.200526 6841 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:58:02.200881 6841 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:02.201195 6841 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:58:02.202852 6345 leader_election.cc:343] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:02.210954 6422 tablet_bootstrap.cc:654] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:02.217239 6841 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.222149 6841 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:02.223002 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:02.226836 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:58:02.227005 6345 leader_election.cc:343] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Illegal state: must be running to vote when last-logged opid is not known
I20260620 19:58:02.227506 6301 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:02.237522 6422 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:02.237581 6301 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:02.238307 6422 ts_tablet_manager.cc:1403] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.010s
I20260620 19:58:02.239478 6344 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:02.240779 6841 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:58:02.241233 6841 raft_consensus.cc:697] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:02.241169 6422 raft_consensus.cc:359] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.242028 6422 raft_consensus.cc:385] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:02.242424 6422 raft_consensus.cc:740] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:02.242100 6841 consensus_queue.cc:237] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.243678 6422 consensus_queue.cc:260] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.246044 6422 ts_tablet_manager.cc:1434] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:58:02.247112 6422 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:02.248647 6130 catalog_manager.cc:5696] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:02.250264 6130 catalog_manager.cc:5696] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:02.253012 6422 tablet_bootstrap.cc:654] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:02.257968 6422 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:02.258392 6422 ts_tablet_manager.cc:1403] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 19:58:02.260775 6422 raft_consensus.cc:359] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.261415 6422 raft_consensus.cc:385] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:02.261685 6422 raft_consensus.cc:740] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:02.262346 6422 consensus_queue.cc:260] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.264226 6422 ts_tablet_manager.cc:1434] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:58:02.265057 6422 tablet_bootstrap.cc:492] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6: Bootstrap starting.
I20260620 19:58:02.269724 6422 tablet_bootstrap.cc:654] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6: Neither blocks nor log segments found. Creating new log.
I20260620 19:58:02.274138 6422 tablet_bootstrap.cc:492] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6: No bootstrap required, opened a new log
I20260620 19:58:02.274533 6422 ts_tablet_manager.cc:1403] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 19:58:02.277772 6422 raft_consensus.cc:359] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.278385 6422 raft_consensus.cc:385] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:58:02.278654 6422 raft_consensus.cc:740] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Initialized, Role: FOLLOWER
I20260620 19:58:02.279238 6422 consensus_queue.cc:260] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.279611 6130 catalog_manager.cc:5696] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:02.281229 6422 ts_tablet_manager.cc:1434] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260620 19:58:02.286849 6750 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:02.304222 6750 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:02.311465 6841 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:02.316242 6645 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:02.323495 6230 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:02.399844 6843 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:02.400336 6843 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.402400 6843 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:02.403034 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:02.403417 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:02.405316 6195 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:02.405928 6750 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:02.410279 6740 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:02.410774 6740 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.412671 6740 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:02.412962 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:02.413617 6230 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:02.413673 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:02.414254 6376 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:02.414518 6271 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:02.415337 6740 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:02.415731 6740 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:02.416124 6740 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:02.419351 6841 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:02.424701 6740 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.428218 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:02.428730 6376 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:02.429021 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:02.429665 6230 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:02.430225 6225 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:02.432989 6740 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:02.438831 6230 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:02.440133 6271 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:02.442557 6842 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:02.443361 6842 raft_consensus.cc:697] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:02.444940 6841 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 19:58:02.446511 6842 consensus_queue.cc:237] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.449882 6376 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:02.465849 6128 catalog_manager.cc:5696] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 0 to 2, leader changed from <none> to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:02.612812 6843 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:02.613262 6843 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:02.615026 6843 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:02.615598 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:02.615801 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:02.616463 6299 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 1: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:02.617312 6195 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:02.617897 6843 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:02.623381 6843 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:58:02.653317 6230 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:58:02.698055 6841 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:02.713586 6645 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:03.069547 6230 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260620 19:58:03.070576 6842 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:03.074117 6377 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260620 19:58:03.075654 6842 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:58:03.702229 5074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260620 19:58:03.741266 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:58:03.741804 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:58:03.741117 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:58:03.744555 5074 server_base.cc:1061] running on GCE node
I20260620 19:58:03.745942 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260620 19:58:03.746274 5074 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260620 19:58:03.746544 5074 hybrid_clock.cc:648] HybridClock initialized: now 1781985483746523 us; error 0 us; skew 500 ppm
I20260620 19:58:03.747586 5074 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 19:58:03.751112 5074 webserver.cc:533] Webserver started at http://127.4.244.132:36753/ using document root <none> and password file <none>
I20260620 19:58:03.751681 5074 fs_manager.cc:362] Metadata directory not provided
I20260620 19:58:03.751919 5074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:58:03.752233 5074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:58:03.753414 5074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-3-root/instance:
uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
format_stamp: "Formatted at 2026-06-20 19:58:03 on dist-test-slave-zxgc"
I20260620 19:58:03.758348 5074 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 19:58:03.762116 6873 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:58:03.762951 5074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:58:03.763294 5074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-3-root
uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
format_stamp: "Formatted at 2026-06-20 19:58:03 on dist-test-slave-zxgc"
I20260620 19:58:03.763674 5074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:58:03.777783 5074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260620 19:58:03.779101 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:58:03.780892 5074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260620 19:58:03.784040 5074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:58:03.784296 5074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:58:03.784591 5074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260620 19:58:03.784813 5074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:58:03.848585 5074 rpc_server.cc:307] RPC server started. Bound to: 127.4.244.132:42325
I20260620 19:58:03.848701 6935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.244.132:42325 every 8 connection(s)
I20260620 19:58:03.853287 5074 auto_leader_rebalancer-test.cc:439] add a tserver: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:03.865154 6936 heartbeater.cc:344] Connected to a master server at 127.4.244.190:33963
I20260620 19:58:03.865573 6936 heartbeater.cc:461] Registering TS with master...
I20260620 19:58:03.866472 6936 heartbeater.cc:507] Master 127.4.244.190:33963 requested a full tablet report, sending...
I20260620 19:58:03.868348 6128 ts_manager.cc:194] Registered new tserver with Master: 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:03.869640 6128 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45406
I20260620 19:58:04.009446 6230 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.017208 6377 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:04.017781 6299 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:04.019114 6747 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:04.020282 6843 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:04.032418 6843 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.034186 6376 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.037171 6305 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:04.048514 6196 consensus_peers.cc:597] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c06fb2d65824956840c8faf71df4cc9. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:04.074891 6128 catalog_manager.cc:5696] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.106976 6376 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.119985 6305 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:04.121447 6825 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:04.126286 6230 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:04.127740 6825 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:04.154042 6230 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.149142 6954 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.171772 6130 catalog_manager.cc:5696] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:04.177029 6345 consensus_peers.cc:597] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97eee360f90e4475858b88d88f8efc5c. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:04.182454 6301 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.186875 6376 consensus_queue.cc:237] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.195843 6301 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:04.196170 6345 consensus_peers.cc:597] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 779819d94546427fa367040185e417ab. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:04.196370 6230 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:04.197259 6495 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:04.198186 6954 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:04.210687 6825 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.215492 6305 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.260257 6130 catalog_manager.cc:5696] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:04.271428 6226 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.275205 6305 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.293534 6376 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:04.295868 6948 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:04.302146 6226 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:04.309348 6965 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:04.313221 6948 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.314821 6377 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:04.316356 6271 consensus_peers.cc:597] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a506f4812fdc4ae2b57cb125ac69acb6. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:04.326876 6226 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.333773 6130 catalog_manager.cc:5696] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:04.581485 6968 ts_tablet_manager.cc:933] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:04.583346 6968 tablet_copy_client.cc:323] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:04.597712 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 1c06fb2d65824956840c8faf71df4cc9 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:04.598289 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 1c06fb2d65824956840c8faf71df4cc9 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-1c06fb2d65824956840c8faf71df4cc9
I20260620 19:58:04.604417 6242 tablet_copy_source_session.cc:215] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:04.607856 6968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c06fb2d65824956840c8faf71df4cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:04.626147 6968 tablet_copy_client.cc:806] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:04.626914 6968 tablet_copy_client.cc:670] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:04.632995 6968 tablet_copy_client.cc:538] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:04.644441 6968 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:04.670856 6968 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:04.672067 6968 tablet_bootstrap.cc:492] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:04.672739 6968 ts_tablet_manager.cc:1403] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.013s
I20260620 19:58:04.672942 6973 ts_tablet_manager.cc:933] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:04.675266 6973 tablet_copy_client.cc:323] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:04.675525 6968 raft_consensus.cc:359] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.676477 6968 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:04.679692 6968 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.687163 6936 heartbeater.cc:499] Master 127.4.244.190:33963 was elected leader, sending a full tablet report...
I20260620 19:58:04.687587 6968 ts_tablet_manager.cc:1434] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.015s user 0.010s sys 0.001s
I20260620 19:58:04.691015 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-1c06fb2d65824956840c8faf71df4cc9 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:04.691639 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-1c06fb2d65824956840c8faf71df4cc9 on tablet 1c06fb2d65824956840c8faf71df4cc9 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:04.700390 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 779819d94546427fa367040185e417ab from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:04.701102 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 779819d94546427fa367040185e417ab from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-779819d94546427fa367040185e417ab
I20260620 19:58:04.703153 6968 ts_tablet_manager.cc:933] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:04.723104 6978 ts_tablet_manager.cc:933] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:04.713469 6391 tablet_copy_source_session.cc:215] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:04.731907 6973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779819d94546427fa367040185e417ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:04.740789 6968 tablet_copy_client.cc:323] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:04.741182 6978 tablet_copy_client.cc:323] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:04.744673 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 97eee360f90e4475858b88d88f8efc5c from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:04.745345 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 97eee360f90e4475858b88d88f8efc5c from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-97eee360f90e4475858b88d88f8efc5c
I20260620 19:58:04.750337 6973 tablet_copy_client.cc:806] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:04.751088 6973 tablet_copy_client.cc:670] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:04.754995 6391 tablet_copy_source_session.cc:215] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:04.759634 6978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97eee360f90e4475858b88d88f8efc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:04.762077 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet a506f4812fdc4ae2b57cb125ac69acb6 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:04.762730 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet a506f4812fdc4ae2b57cb125ac69acb6 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-a506f4812fdc4ae2b57cb125ac69acb6
I20260620 19:58:04.765606 6973 tablet_copy_client.cc:538] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:04.770257 6317 tablet_copy_source_session.cc:215] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:04.776309 6973 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:04.781872 6968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a506f4812fdc4ae2b57cb125ac69acb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:04.798794 6968 tablet_copy_client.cc:806] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:04.799664 6968 tablet_copy_client.cc:670] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:04.807996 6968 tablet_copy_client.cc:538] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:04.814394 6978 tablet_copy_client.cc:806] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:04.815172 6978 tablet_copy_client.cc:670] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:04.819123 6968 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:04.827745 6978 tablet_copy_client.cc:538] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:04.844384 6978 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:04.868391 6973 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:04.869416 6973 tablet_bootstrap.cc:492] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:04.870126 6973 ts_tablet_manager.cc:1403] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.094s user 0.027s sys 0.008s
I20260620 19:58:04.884675 6973 raft_consensus.cc:359] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.885660 6973 raft_consensus.cc:740] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:04.886338 6973 consensus_queue.cc:260] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.888837 6968 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:04.889858 6968 tablet_bootstrap.cc:492] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:04.890592 6968 ts_tablet_manager.cc:1403] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.072s user 0.021s sys 0.004s
I20260620 19:58:04.893922 6968 raft_consensus.cc:359] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.895172 6973 ts_tablet_manager.cc:1434] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260620 19:58:04.895124 6968 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:04.896159 6968 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.899825 6968 ts_tablet_manager.cc:1434] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260620 19:58:04.903535 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-779819d94546427fa367040185e417ab received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:04.904042 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-779819d94546427fa367040185e417ab on tablet 779819d94546427fa367040185e417ab with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:04.907667 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a506f4812fdc4ae2b57cb125ac69acb6 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:04.908243 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a506f4812fdc4ae2b57cb125ac69acb6 on tablet a506f4812fdc4ae2b57cb125ac69acb6 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:04.925993 6978 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:04.930240 6978 tablet_bootstrap.cc:492] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:04.930968 6978 ts_tablet_manager.cc:1403] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.087s user 0.016s sys 0.008s
I20260620 19:58:04.934355 6978 raft_consensus.cc:359] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.935216 6978 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:04.935861 6978 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:04.938167 6978 ts_tablet_manager.cc:1434] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260620 19:58:04.939782 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-97eee360f90e4475858b88d88f8efc5c received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:04.940165 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-97eee360f90e4475858b88d88f8efc5c on tablet 97eee360f90e4475858b88d88f8efc5c with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:05.190505 6911 raft_consensus.cc:1217] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:05.264513 6911 raft_consensus.cc:1217] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:05.289171 6911 raft_consensus.cc:1217] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:05.303570 6911 raft_consensus.cc:1217] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:05.675231 6954 raft_consensus.cc:1064] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:05.677927 6954 consensus_queue.cc:237] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:05.684796 6948 raft_consensus.cc:1064] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:05.687388 6911 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.688647 6954 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:05.689911 6226 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.692306 6495 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:05.696661 6299 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.705420 6958 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:05.732373 6958 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.737749 6980 raft_consensus.cc:1064] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:05.740231 6980 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:05.751175 6226 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.755887 6299 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.759814 6129 catalog_manager.cc:5696] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:05.764734 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 73896.2us
I20260620 19:58:05.767160 6948 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:05.774618 6911 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.792443 6910 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.794626 6843 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:05.800977 6376 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.802491 6980 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:05.819309 6911 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.821656 6842 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:05.824252 6226 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.827220 6948 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:05.841110 6377 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.843367 6299 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:05.847860 6965 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.849107 6226 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.848562 6948 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260620 19:58:05.845839 6980 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260620 19:58:05.867777 7001 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.877171 6911 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.885771 6377 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.892549 6376 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.891772 6128 catalog_manager.cc:5696] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:05.907513 6375 consensus_queue.cc:237] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:05.921540 6825 raft_consensus.cc:1064] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:05.927462 6825 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:05.931250 6910 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.944751 6911 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:05.946450 6227 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:05.946091 6495 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:05.949695 6301 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:05.957556 6958 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:05.960547 6909 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:05.962766 6965 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:05.968703 6126 catalog_manager.cc:5696] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:05.970458 6948 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.977211 6909 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:05.996546 6226 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:06.004133 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet a506f4812fdc4ae2b57cb125ac69acb6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:06.010376 6495 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.013340 6985 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:06.024971 6225 raft_consensus.cc:2955] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.028822 6128 catalog_manager.cc:5696] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:58:05.987150 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 52634.7us
I20260620 19:58:06.083828 6305 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:58:06.072922 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 53416.4us
I20260620 19:58:06.096215 6909 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.098831 6225 raft_consensus.cc:2955] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.120008 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 779819d94546427fa367040185e417ab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:06.129673 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet a506f4812fdc4ae2b57cb125ac69acb6 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:06.131193 6227 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:06.137141 6909 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:06.138571 6825 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:06.141117 6299 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:06.141042 6825 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:06.143671 6127 catalog_manager.cc:5696] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:06.144822 6825 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:06.162837 6227 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:06.167618 6911 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:06.169279 6980 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:06.173002 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 779819d94546427fa367040185e417ab with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:06.176802 6301 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:06.178750 7001 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:06.180559 7001 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.184005 7009 tablet_replica.cc:333] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:06.185309 6909 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.192827 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 1c06fb2d65824956840c8faf71df4cc9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:06.193696 7009 raft_consensus.cc:2243] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:06.194343 7009 raft_consensus.cc:2272] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:06.195493 6301 raft_consensus.cc:2955] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.199124 7012 tablet_replica.cc:333] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:06.207584 7012 raft_consensus.cc:2243] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:06.208283 7012 raft_consensus.cc:2272] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:06.209877 6129 catalog_manager.cc:5696] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.212782 7012 ts_tablet_manager.cc:1916] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:06.232437 6299 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.233954 6495 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.236181 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c06fb2d65824956840c8faf71df4cc9 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:06.248045 6225 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.260695 6127 catalog_manager.cc:5696] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.264467 6909 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.264495 7012 ts_tablet_manager.cc:1929] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:06.271566 7012 log.cc:1199] T 779819d94546427fa367040185e417ab P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/779819d94546427fa367040185e417ab
I20260620 19:58:06.274191 7009 ts_tablet_manager.cc:1916] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:06.275619 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 779819d94546427fa367040185e417ab (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:06.292919 7009 ts_tablet_manager.cc:1929] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:06.293401 7009 log.cc:1199] T a506f4812fdc4ae2b57cb125ac69acb6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/a506f4812fdc4ae2b57cb125ac69acb6
I20260620 19:58:06.295099 7009 tablet_replica.cc:333] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:06.295897 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet a506f4812fdc4ae2b57cb125ac69acb6 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:06.296089 7009 raft_consensus.cc:2243] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:06.296856 7009 raft_consensus.cc:2272] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:06.293006 6377 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:06.300585 7009 ts_tablet_manager.cc:1916] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:06.301721 6909 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:06.303133 6825 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:06.302577 6299 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:06.307512 6825 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:06.310880 7005 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.320515 7009 ts_tablet_manager.cc:1929] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:06.321046 7009 log.cc:1199] T 1c06fb2d65824956840c8faf71df4cc9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/1c06fb2d65824956840c8faf71df4cc9
I20260620 19:58:06.321376 6909 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.322943 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 1c06fb2d65824956840c8faf71df4cc9 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:06.332895 6305 raft_consensus.cc:2955] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.338191 6128 catalog_manager.cc:5696] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:06.342820 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 97eee360f90e4475858b88d88f8efc5c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:06.359146 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet 97eee360f90e4475858b88d88f8efc5c with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:06.362635 7014 tablet_replica.cc:333] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:06.363637 7014 raft_consensus.cc:2243] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:06.364321 7014 raft_consensus.cc:2272] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:06.367233 7014 ts_tablet_manager.cc:1916] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:06.380393 7014 ts_tablet_manager.cc:1929] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:06.380662 7014 log.cc:1199] T 97eee360f90e4475858b88d88f8efc5c P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/97eee360f90e4475858b88d88f8efc5c
I20260620 19:58:06.381965 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet 97eee360f90e4475858b88d88f8efc5c (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:07.423970 6305 consensus_queue.cc:237] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:07.429059 6377 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:07.429533 6271 consensus_peers.cc:597] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ace7015fc514a909c3139c9e717b387. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:07.430409 7008 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:07.430910 6225 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:07.432160 7008 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:07.443526 7007 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.460484 6227 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:07.467760 6377 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.468773 6129 catalog_manager.cc:5696] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:07.478097 6196 consensus_peers.cc:597] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cb7a18e80e145f6913b635487750934. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:07.479962 6305 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:07.480350 6227 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.483600 6381 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:07.487844 6980 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:07.496219 6980 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:07.520057 7015 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.527038 6375 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.528811 6299 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.538640 6377 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:07.547938 6345 consensus_peers.cc:597] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43e3501e8f31428cb18618fc6a8e1f59. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:07.548049 6299 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:07.550290 7005 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:07.550731 6225 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:07.552417 7004 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:07.575879 7005 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.580286 6128 catalog_manager.cc:5696] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.582515 6305 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.597867 6299 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:07.604640 6227 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:07.637355 6271 consensus_peers.cc:597] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5c97b15b2f74ff2a06fde66e407b188. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:07.641536 6377 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:07.642776 7022 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:07.647042 6225 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:07.651391 7008 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:07.677153 7020 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:07.821137 6377 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:08.417377 6198 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.009s sys 0.013s
W20260620 19:58:08.417820 6198 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.009s sys 0.013s
I20260620 19:58:08.421608 7054 ts_tablet_manager.cc:933] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
W20260620 19:58:08.422509 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 65092us
I20260620 19:58:08.440202 6129 catalog_manager.cc:5696] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:08.462661 6129 catalog_manager.cc:5696] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:08.467826 7058 ts_tablet_manager.cc:933] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:08.485002 7058 tablet_copy_client.cc:323] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:08.509033 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 5ace7015fc514a909c3139c9e717b387 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:08.509689 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 5ace7015fc514a909c3139c9e717b387 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-5ace7015fc514a909c3139c9e717b387
I20260620 19:58:08.518517 6317 tablet_copy_source_session.cc:215] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:08.527180 6224 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:08.529887 7058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ace7015fc514a909c3139c9e717b387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:08.552456 7057 ts_tablet_manager.cc:933] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:08.572815 7056 ts_tablet_manager.cc:933] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:08.580952 7058 tablet_copy_client.cc:806] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:08.581903 7058 tablet_copy_client.cc:670] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:08.587659 7057 tablet_copy_client.cc:323] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:08.596671 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet c5c97b15b2f74ff2a06fde66e407b188 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:08.597162 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet c5c97b15b2f74ff2a06fde66e407b188 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-c5c97b15b2f74ff2a06fde66e407b188
I20260620 19:58:08.592361 7054 tablet_copy_client.cc:323] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:08.604240 7056 tablet_copy_client.cc:323] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:08.605759 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 43e3501e8f31428cb18618fc6a8e1f59 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:08.606233 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 43e3501e8f31428cb18618fc6a8e1f59 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-43e3501e8f31428cb18618fc6a8e1f59
I20260620 19:58:08.609872 7058 tablet_copy_client.cc:538] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:08.609779 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 1cb7a18e80e145f6913b635487750934 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:08.610386 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 1cb7a18e80e145f6913b635487750934 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-1cb7a18e80e145f6913b635487750934
I20260620 19:58:08.613812 6391 tablet_copy_source_session.cc:215] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:08.616757 7056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43e3501e8f31428cb18618fc6a8e1f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:08.618228 6242 tablet_copy_source_session.cc:215] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:08.609799 6317 tablet_copy_source_session.cc:215] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:08.622438 7054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb7a18e80e145f6913b635487750934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:08.623183 7057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c97b15b2f74ff2a06fde66e407b188. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:08.630124 7058 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:08.642355 7054 tablet_copy_client.cc:806] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:08.642999 7054 tablet_copy_client.cc:670] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:08.643621 7057 tablet_copy_client.cc:806] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:08.644219 7057 tablet_copy_client.cc:670] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:08.646700 7056 tablet_copy_client.cc:806] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:08.647228 7056 tablet_copy_client.cc:670] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:08.648375 7054 tablet_copy_client.cc:538] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:08.650389 7057 tablet_copy_client.cc:538] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:08.650830 7056 tablet_copy_client.cc:538] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:08.660888 7056 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:08.665709 7054 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:08.672224 7057 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:08.697872 7058 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:08.700932 7058 tablet_bootstrap.cc:492] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:08.702953 7058 ts_tablet_manager.cc:1403] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.073s user 0.020s sys 0.009s
I20260620 19:58:08.714841 7057 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:08.720144 7058 raft_consensus.cc:359] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.726430 7058 raft_consensus.cc:740] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:08.726876 7057 tablet_bootstrap.cc:492] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:08.727216 7058 consensus_queue.cc:260] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.722457 7054 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:08.728852 7054 tablet_bootstrap.cc:492] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:08.730783 7054 ts_tablet_manager.cc:1403] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.065s user 0.030s sys 0.002s
I20260620 19:58:08.721690 7056 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:08.734269 7056 tablet_bootstrap.cc:492] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:08.727881 7057 ts_tablet_manager.cc:1403] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.056s user 0.018s sys 0.005s
I20260620 19:58:08.737241 7056 ts_tablet_manager.cc:1403] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.077s user 0.023s sys 0.000s
I20260620 19:58:08.740197 7054 raft_consensus.cc:359] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.743108 7054 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:08.745460 7054 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.744390 7056 raft_consensus.cc:359] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.747639 7056 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:08.748322 7056 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.750692 7056 ts_tablet_manager.cc:1434] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.013s user 0.003s sys 0.004s
I20260620 19:58:08.742043 7057 raft_consensus.cc:359] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.751077 7057 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:08.751713 7057 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:08.745064 7058 ts_tablet_manager.cc:1434] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.042s user 0.008s sys 0.000s
I20260620 19:58:08.754853 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-43e3501e8f31428cb18618fc6a8e1f59 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:08.756930 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-43e3501e8f31428cb18618fc6a8e1f59 on tablet 43e3501e8f31428cb18618fc6a8e1f59 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:08.757682 7054 ts_tablet_manager.cc:1434] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.027s user 0.007s sys 0.003s
I20260620 19:58:08.759969 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-1cb7a18e80e145f6913b635487750934 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:08.760396 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-1cb7a18e80e145f6913b635487750934 on tablet 1cb7a18e80e145f6913b635487750934 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:08.763464 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-5ace7015fc514a909c3139c9e717b387 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:08.763895 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-5ace7015fc514a909c3139c9e717b387 on tablet 5ace7015fc514a909c3139c9e717b387 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:58:08.772799 6161 debug-util.cc:398] Leaking SignalData structure 0x7b0800035720 after lost signal to thread 5077
W20260620 19:58:08.773857 6161 debug-util.cc:398] Leaking SignalData structure 0x7b0800199c40 after lost signal to thread 6164
W20260620 19:58:08.775069 6161 debug-util.cc:398] Leaking SignalData structure 0x7b080021a220 after lost signal to thread 6256
W20260620 19:58:08.776129 6161 debug-util.cc:398] Leaking SignalData structure 0x7b0800307500 after lost signal to thread 6331
W20260620 19:58:08.777114 6161 debug-util.cc:398] Leaking SignalData structure 0x7b0800290be0 after lost signal to thread 6405
I20260620 19:58:08.777837 7057 ts_tablet_manager.cc:1434] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.043s user 0.009s sys 0.001s
W20260620 19:58:08.778159 6161 debug-util.cc:398] Leaking SignalData structure 0x7b080031c320 after lost signal to thread 6935
I20260620 19:58:08.780035 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-c5c97b15b2f74ff2a06fde66e407b188 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:08.780460 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-c5c97b15b2f74ff2a06fde66e407b188 on tablet c5c97b15b2f74ff2a06fde66e407b188 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:08.849756 6911 raft_consensus.cc:1217] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:08.905547 6911 raft_consensus.cc:1217] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:09.024375 6911 raft_consensus.cc:1217] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:09.252774 6911 raft_consensus.cc:1217] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:09.434849 7050 raft_consensus.cc:1064] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:09.438412 7050 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:09.465176 7065 raft_consensus.cc:1064] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:09.472703 7065 consensus_queue.cc:237] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:09.480376 6911 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.481691 7004 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.483803 6226 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.485532 7004 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.486738 6301 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.501567 7050 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.528237 6908 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.529294 6226 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.532836 7019 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.534444 6375 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.535885 7019 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260620 19:58:09.542631 7019 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:09.546825 7021 raft_consensus.cc:1064] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:09.550746 7021 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:09.564608 7048 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.566511 6230 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.567637 6909 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.568801 6910 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.582814 6375 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.585062 7019 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.585436 6129 catalog_manager.cc:5696] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.586666 6301 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.600368 7019 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.611718 6227 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.611896 7019 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.630887 7021 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:09.640770 6375 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.665719 6230 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.697229 7075 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.711800 6377 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.723213 6910 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.758415 6230 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.768505 6911 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.844192 6130 catalog_manager.cc:5696] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:09.849102 6377 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:09.857568 7026 raft_consensus.cc:1064] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:09.860065 7026 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:09.866446 6911 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:09.869391 7050 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:09.869851 6129 catalog_manager.cc:5696] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.878731 6301 consensus_queue.cc:237] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:09.882736 6230 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:09.884496 7048 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:09.885035 6910 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.886531 7073 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.887240 7048 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.888921 6910 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.895076 6375 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:09.896692 7021 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:09.900513 6908 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:09.902983 7090 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:09.905241 7092 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.918942 6129 catalog_manager.cc:5696] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.921012 6307 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.923084 6908 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.918375 6375 raft_consensus.cc:2955] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.943517 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 43e3501e8f31428cb18618fc6a8e1f59 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:09.944685 6377 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:09.956745 6980 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.957762 7073 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:09.951000 6301 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:09.971737 6227 raft_consensus.cc:2955] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:09.999747 6911 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:10.002166 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 43e3501e8f31428cb18618fc6a8e1f59 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:10.010665 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 5ace7015fc514a909c3139c9e717b387 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:10.013469 7065 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:09.996766 6127 catalog_manager.cc:5696] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:10.029277 7090 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.035138 6910 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.046721 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet 5ace7015fc514a909c3139c9e717b387 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:10.058983 6375 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:10.060731 7097 tablet_replica.cc:333] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:10.061017 7065 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:10.060966 7026 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.063381 7097 raft_consensus.cc:2243] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:10.065114 7097 raft_consensus.cc:2272] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:10.069955 7097 ts_tablet_manager.cc:1916] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260620 19:58:10.062304 7098 log.cc:927] Time spent T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260620 19:58:10.064853 6305 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.072484 7101 tablet_replica.cc:333] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:10.077618 7101 raft_consensus.cc:2243] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:10.080407 7101 raft_consensus.cc:2272] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:10.083885 7101 ts_tablet_manager.cc:1916] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:10.082638 6908 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.102996 6375 raft_consensus.cc:2955] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.106096 7097 ts_tablet_manager.cc:1929] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:10.107402 7097 log.cc:1199] T 43e3501e8f31428cb18618fc6a8e1f59 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/43e3501e8f31428cb18618fc6a8e1f59
I20260620 19:58:10.110180 6377 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.116050 6127 catalog_manager.cc:5696] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.117435 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet c5c97b15b2f74ff2a06fde66e407b188 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:10.139936 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet c5c97b15b2f74ff2a06fde66e407b188 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:10.117642 6129 catalog_manager.cc:5696] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:10.144424 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 43e3501e8f31428cb18618fc6a8e1f59 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:10.148095 7101 ts_tablet_manager.cc:1929] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:10.148695 7101 log.cc:1199] T 5ace7015fc514a909c3139c9e717b387 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/5ace7015fc514a909c3139c9e717b387
I20260620 19:58:10.151085 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet 5ace7015fc514a909c3139c9e717b387 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:10.167349 7101 tablet_replica.cc:333] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:10.168569 7101 raft_consensus.cc:2243] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:10.169245 7101 raft_consensus.cc:2272] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:10.174593 6224 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:10.184235 6908 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:10.186082 6375 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:10.186101 6980 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:10.187934 7073 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:10.195124 7026 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.197520 7101 ts_tablet_manager.cc:1916] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:10.201169 6908 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.207391 6375 raft_consensus.cc:2955] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.241576 7101 ts_tablet_manager.cc:1929] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:10.242053 7101 log.cc:1199] T c5c97b15b2f74ff2a06fde66e407b188 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/c5c97b15b2f74ff2a06fde66e407b188
I20260620 19:58:10.243741 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet c5c97b15b2f74ff2a06fde66e407b188 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:10.255466 6127 catalog_manager.cc:5696] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:10.262087 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 1cb7a18e80e145f6913b635487750934 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:10.285741 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cb7a18e80e145f6913b635487750934 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:10.286944 7097 tablet_replica.cc:333] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:10.287950 7097 raft_consensus.cc:2243] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:10.288691 7097 raft_consensus.cc:2272] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:10.292788 7097 ts_tablet_manager.cc:1916] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:10.314481 7097 ts_tablet_manager.cc:1929] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:10.314944 7097 log.cc:1199] T 1cb7a18e80e145f6913b635487750934 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/1cb7a18e80e145f6913b635487750934
I20260620 19:58:10.316715 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 1cb7a18e80e145f6913b635487750934 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:11.378453 6307 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:11.387253 6376 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:11.387342 6271 consensus_peers.cc:597] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 758c2974c107466bbcc6c41d00de7b18. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:11.388868 7099 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:11.394287 6224 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:11.396168 7099 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:11.401194 7090 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.403524 6376 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.407378 6227 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.417035 6376 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:11.420009 6127 catalog_manager.cc:5696] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:11.449889 6345 consensus_peers.cc:597] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1858fc28e4948cc8e3c29e06357d166. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:11.452862 6307 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:11.454842 6227 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:11.454875 7035 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:11.464531 7102 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:11.488495 7035 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.493988 6227 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.496337 6305 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.518503 6129 catalog_manager.cc:5696] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.525928 6377 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:11.579173 6227 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:11.589272 6301 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:11.590953 7102 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:11.596099 6345 consensus_peers.cc:597] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db666bffcf904fa59d91db75dc91c8b0. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:11.596601 7035 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:11.606828 7102 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.618660 6305 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.620625 6227 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.633159 6301 consensus_queue.cc:237] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:11.637527 6129 catalog_manager.cc:5696] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:11.672947 6224 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:11.674400 7090 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:11.675887 6377 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:11.678423 6271 consensus_peers.cc:597] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8504518fd53942459d685f3abcbae2eb. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:11.682710 7065 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:11.710883 7065 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.712512 6377 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.764353 6129 catalog_manager.cc:5696] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:11.873461 6224 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:12.046731 7126 ts_tablet_manager.cc:933] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:12.049233 7127 ts_tablet_manager.cc:933] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:12.049954 7129 ts_tablet_manager.cc:933] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:12.093014 7127 tablet_copy_client.cc:323] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:12.095331 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet a1858fc28e4948cc8e3c29e06357d166 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:12.096208 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet a1858fc28e4948cc8e3c29e06357d166 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-a1858fc28e4948cc8e3c29e06357d166
I20260620 19:58:12.093065 7129 tablet_copy_client.cc:323] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:12.100049 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 758c2974c107466bbcc6c41d00de7b18 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:12.100606 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 758c2974c107466bbcc6c41d00de7b18 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-758c2974c107466bbcc6c41d00de7b18
I20260620 19:58:12.106356 7126 tablet_copy_client.cc:323] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:12.108355 6390 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet db666bffcf904fa59d91db75dc91c8b0 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:12.108886 6390 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet db666bffcf904fa59d91db75dc91c8b0 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-db666bffcf904fa59d91db75dc91c8b0
I20260620 19:58:12.122130 6390 tablet_copy_source_session.cc:215] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:12.126662 7126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db666bffcf904fa59d91db75dc91c8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:12.141554 6391 tablet_copy_source_session.cc:215] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:12.145221 7127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1858fc28e4948cc8e3c29e06357d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:12.152449 7126 tablet_copy_client.cc:806] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:12.153105 7126 tablet_copy_client.cc:670] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:12.136088 6317 tablet_copy_source_session.cc:215] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:12.162045 7129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758c2974c107466bbcc6c41d00de7b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:12.171238 7126 tablet_copy_client.cc:538] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:12.193980 7127 tablet_copy_client.cc:806] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:12.199542 7127 tablet_copy_client.cc:670] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:12.203362 7129 tablet_copy_client.cc:806] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:12.206707 7129 tablet_copy_client.cc:670] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:12.209533 7127 tablet_copy_client.cc:538] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:12.195868 7126 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:12.218765 7129 tablet_copy_client.cc:538] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:12.243665 7127 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:12.246182 7129 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:12.260298 7133 ts_tablet_manager.cc:933] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:12.259960 7126 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:12.261504 7126 tablet_bootstrap.cc:492] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:12.264823 7126 ts_tablet_manager.cc:1403] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.018s
I20260620 19:58:12.268229 7126 raft_consensus.cc:359] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.269047 7126 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:12.269708 7126 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.284508 7133 tablet_copy_client.cc:323] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:12.293970 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 8504518fd53942459d685f3abcbae2eb from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:12.296939 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 8504518fd53942459d685f3abcbae2eb from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-8504518fd53942459d685f3abcbae2eb
I20260620 19:58:12.312659 6317 tablet_copy_source_session.cc:215] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:12.314193 7127 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:12.316846 7133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8504518fd53942459d685f3abcbae2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:12.318626 7126 ts_tablet_manager.cc:1434] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.053s user 0.022s sys 0.014s
I20260620 19:58:12.320371 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-db666bffcf904fa59d91db75dc91c8b0 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:12.320842 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-db666bffcf904fa59d91db75dc91c8b0 on tablet db666bffcf904fa59d91db75dc91c8b0 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:12.343254 7127 tablet_bootstrap.cc:492] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:12.344055 7127 ts_tablet_manager.cc:1403] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.101s user 0.023s sys 0.014s
I20260620 19:58:12.351765 7127 raft_consensus.cc:359] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.352614 7127 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:12.357720 7129 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:12.358796 7129 tablet_bootstrap.cc:492] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:12.359544 7129 ts_tablet_manager.cc:1403] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.114s user 0.026s sys 0.009s
I20260620 19:58:12.362499 7129 raft_consensus.cc:359] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.353299 7127 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.367326 7133 tablet_copy_client.cc:806] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:12.375228 7133 tablet_copy_client.cc:670] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:12.375696 7127 ts_tablet_manager.cc:1434] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.031s user 0.003s sys 0.004s
I20260620 19:58:12.376550 7129 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:12.377327 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a1858fc28e4948cc8e3c29e06357d166 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:12.377790 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a1858fc28e4948cc8e3c29e06357d166 on tablet a1858fc28e4948cc8e3c29e06357d166 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:12.382947 7129 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.385725 7129 ts_tablet_manager.cc:1434] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260620 19:58:12.387565 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-758c2974c107466bbcc6c41d00de7b18 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:12.388005 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-758c2974c107466bbcc6c41d00de7b18 on tablet 758c2974c107466bbcc6c41d00de7b18 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:12.393281 7133 tablet_copy_client.cc:538] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:12.404162 7133 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:12.445480 7133 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:12.446411 7133 tablet_bootstrap.cc:492] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:12.449982 7133 ts_tablet_manager.cc:1403] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.046s user 0.031s sys 0.011s
I20260620 19:58:12.451737 6908 raft_consensus.cc:1217] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:12.458962 7133 raft_consensus.cc:359] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.459969 7133 raft_consensus.cc:740] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:12.460776 7133 consensus_queue.cc:260] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:12.465163 7133 ts_tablet_manager.cc:1434] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260620 19:58:12.468407 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8504518fd53942459d685f3abcbae2eb received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:12.468837 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8504518fd53942459d685f3abcbae2eb on tablet 8504518fd53942459d685f3abcbae2eb with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:12.647267 6908 raft_consensus.cc:1217] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:12.662235 6908 raft_consensus.cc:1217] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:12.784228 6908 raft_consensus.cc:1217] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:13.014510 7102 raft_consensus.cc:1064] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:13.017027 7102 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.043743 6908 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.046218 6224 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.048882 6301 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.053365 7103 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.056655 7128 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:13.063409 7103 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.089321 7102 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.095278 6908 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.102299 6224 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.111469 6127 catalog_manager.cc:5696] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:13.133893 6301 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.172542 7103 raft_consensus.cc:1064] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:13.177973 7103 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.202327 6908 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.204006 6301 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.208024 6224 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.208844 7102 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.210069 7103 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.213197 7128 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.245481 7121 raft_consensus.cc:1064] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:13.244720 7102 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.247862 7121 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.254248 6305 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.267961 7065 raft_consensus.cc:1064] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:13.269472 6908 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.271888 7065 consensus_queue.cc:237] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.293807 6910 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.296012 6374 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.298692 7099 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:13.300500 6227 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.301703 7099 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.305850 7099 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.335909 6375 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.348970 6128 catalog_manager.cc:5696] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.356623 6910 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.356809 6231 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.364162 7157 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:13.369664 6231 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.370118 7121 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:13.371263 6225 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.372702 6910 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.374439 6377 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:13.375564 7148 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:13.379138 7148 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:13.382148 7099 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:13.394456 7128 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.399513 7154 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.402416 6910 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.407213 6225 raft_consensus.cc:2955] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.402717 6227 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.420151 6908 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.424144 6375 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.432822 6911 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.436830 7103 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:13.437256 6127 catalog_manager.cc:5696] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.438580 6305 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.444801 7103 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:13.451428 7148 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.461068 6909 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.467229 6301 raft_consensus.cc:2955] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.468472 6130 catalog_manager.cc:5696] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.467972 7157 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.477334 6378 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.479029 6224 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.482285 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet a1858fc28e4948cc8e3c29e06357d166 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:13.488917 6128 catalog_manager.cc:5696] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.491508 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet a1858fc28e4948cc8e3c29e06357d166 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:13.511780 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet db666bffcf904fa59d91db75dc91c8b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:13.517733 6374 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:58:13.526110 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 65188.5us
I20260620 19:58:13.526779 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet db666bffcf904fa59d91db75dc91c8b0 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:13.545178 7165 tablet_replica.cc:333] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:13.547147 7165 raft_consensus.cc:2243] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:13.547930 7165 raft_consensus.cc:2272] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:13.552217 7166 tablet_replica.cc:333] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:13.557413 7165 ts_tablet_manager.cc:1916] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:13.553526 7166 raft_consensus.cc:2243] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:13.566753 7166 raft_consensus.cc:2272] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:13.573047 6908 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.578835 6307 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.581280 7165 ts_tablet_manager.cc:1929] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:13.582048 7165 log.cc:1199] T a1858fc28e4948cc8e3c29e06357d166 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/a1858fc28e4948cc8e3c29e06357d166
I20260620 19:58:13.583985 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet a1858fc28e4948cc8e3c29e06357d166 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:13.587493 6127 catalog_manager.cc:5696] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.590143 6909 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.593348 7166 ts_tablet_manager.cc:1916] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:13.594945 7157 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:13.599690 6224 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.605212 7157 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:13.617337 7154 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.626557 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 758c2974c107466bbcc6c41d00de7b18 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:13.633553 6307 consensus_queue.cc:237] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:13.646031 6129 catalog_manager.cc:5696] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:13.649175 6224 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.657763 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 758c2974c107466bbcc6c41d00de7b18 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:13.666254 6231 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.666254 6909 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:13.679241 7167 tablet_replica.cc:333] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:13.673823 6908 raft_consensus.cc:2955] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.681357 7167 raft_consensus.cc:2243] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:13.682057 7167 raft_consensus.cc:2272] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:13.683506 7065 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:13.685734 7167 ts_tablet_manager.cc:1916] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:13.689167 7065 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:13.699848 7166 ts_tablet_manager.cc:1929] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:13.700392 7166 log.cc:1199] T db666bffcf904fa59d91db75dc91c8b0 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/db666bffcf904fa59d91db75dc91c8b0
I20260620 19:58:13.701702 7065 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.704878 6224 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.712095 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet db666bffcf904fa59d91db75dc91c8b0 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:13.716281 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 8504518fd53942459d685f3abcbae2eb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:13.714926 6909 raft_consensus.cc:2955] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:13.717370 7167 ts_tablet_manager.cc:1929] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:13.718021 7167 log.cc:1199] T 758c2974c107466bbcc6c41d00de7b18 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/758c2974c107466bbcc6c41d00de7b18
I20260620 19:58:13.720479 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 758c2974c107466bbcc6c41d00de7b18 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:13.728062 6129 catalog_manager.cc:5696] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:13.747546 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 8504518fd53942459d685f3abcbae2eb with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:13.748807 7167 tablet_replica.cc:333] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:13.749758 7167 raft_consensus.cc:2243] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:13.750447 7167 raft_consensus.cc:2272] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:13.753284 7167 ts_tablet_manager.cc:1916] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:13.771593 7167 ts_tablet_manager.cc:1929] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:13.772015 7167 log.cc:1199] T 8504518fd53942459d685f3abcbae2eb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/8504518fd53942459d685f3abcbae2eb
I20260620 19:58:13.773511 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 8504518fd53942459d685f3abcbae2eb (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:14.821095 6224 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:14.827651 6196 consensus_peers.cc:597] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cf0b0e2dafa4512b82ecefda3013d79. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:14.827821 6378 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:14.828596 6301 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:14.829217 6980 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:14.830044 7169 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:14.840281 7169 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.841892 6301 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.844655 6378 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.859942 6374 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:14.862300 6127 catalog_manager.cc:5696] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:14.879930 6345 consensus_peers.cc:597] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8dba7c3eac4b4392830be760b4060cbc. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:14.880807 6307 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:14.881345 6227 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:14.882174 7102 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:14.883170 7173 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:14.896524 7173 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.907259 6227 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.919386 6307 consensus_queue.cc:237] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:14.939230 6301 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.944011 6130 catalog_manager.cc:5696] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:14.947512 6378 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:14.949635 7065 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:14.965732 6224 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:14.967799 7065 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:14.972285 6271 consensus_peers.cc:597] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc7845798261439d855e526604de104a. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:14.984228 7065 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.993229 6378 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:14.999642 6374 consensus_queue.cc:237] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.003346 6130 catalog_manager.cc:5696] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:15.006914 6224 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:15.017647 6345 consensus_peers.cc:597] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e39bd0f2de444d2acf683f9f18716da. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:15.020854 6227 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:15.021780 6301 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:15.022258 7148 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:15.023303 7182 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:15.031296 7148 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:15.033185 6307 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:15.040485 6227 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:15.054253 6130 catalog_manager.cc:5696] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:15.357020 7191 ts_tablet_manager.cc:933] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:15.358815 7191 tablet_copy_client.cc:323] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:15.360302 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 8dba7c3eac4b4392830be760b4060cbc from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:15.360740 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 8dba7c3eac4b4392830be760b4060cbc from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-8dba7c3eac4b4392830be760b4060cbc
I20260620 19:58:15.366565 6391 tablet_copy_source_session.cc:215] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:15.369608 7191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dba7c3eac4b4392830be760b4060cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:15.389365 7191 tablet_copy_client.cc:806] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:15.390264 7191 tablet_copy_client.cc:670] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:15.392236 7193 ts_tablet_manager.cc:933] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:15.394420 7191 tablet_copy_client.cc:538] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:15.404784 7191 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:15.416502 7193 tablet_copy_client.cc:323] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:15.420272 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet cc7845798261439d855e526604de104a from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:15.420863 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet cc7845798261439d855e526604de104a from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-cc7845798261439d855e526604de104a
I20260620 19:58:15.438370 7191 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:15.439734 7191 tablet_bootstrap.cc:492] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:15.440639 6317 tablet_copy_source_session.cc:215] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:15.441636 7196 ts_tablet_manager.cc:933] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:15.440663 7191 ts_tablet_manager.cc:1403] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.007s
I20260620 19:58:15.444240 7193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7845798261439d855e526604de104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:15.447465 7191 raft_consensus.cc:359] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.453342 7191 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:15.454057 7191 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.455327 7196 tablet_copy_client.cc:323] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:15.457847 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 8cf0b0e2dafa4512b82ecefda3013d79 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:15.458381 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 8cf0b0e2dafa4512b82ecefda3013d79 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-8cf0b0e2dafa4512b82ecefda3013d79
I20260620 19:58:15.466367 6242 tablet_copy_source_session.cc:215] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:15.468699 7191 ts_tablet_manager.cc:1434] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.025s user 0.014s sys 0.003s
I20260620 19:58:15.473440 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8dba7c3eac4b4392830be760b4060cbc received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:15.473901 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8dba7c3eac4b4392830be760b4060cbc on tablet 8dba7c3eac4b4392830be760b4060cbc with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:15.475124 7193 tablet_copy_client.cc:806] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:15.475663 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cf0b0e2dafa4512b82ecefda3013d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:15.475711 7193 tablet_copy_client.cc:670] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:15.482077 7193 tablet_copy_client.cc:538] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:15.491509 7193 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:15.494989 7196 tablet_copy_client.cc:806] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:15.495632 7196 tablet_copy_client.cc:670] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:15.511364 7196 tablet_copy_client.cc:538] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:15.520999 7196 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:15.528151 7193 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:15.529172 7193 tablet_bootstrap.cc:492] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:15.529996 7193 ts_tablet_manager.cc:1403] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.016s
I20260620 19:58:15.533149 7193 raft_consensus.cc:359] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.534027 7193 raft_consensus.cc:740] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:15.534727 7193 consensus_queue.cc:260] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.537242 7193 ts_tablet_manager.cc:1434] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260620 19:58:15.540100 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-cc7845798261439d855e526604de104a received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:15.540555 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-cc7845798261439d855e526604de104a on tablet cc7845798261439d855e526604de104a with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:15.549670 7196 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:15.550715 7196 tablet_bootstrap.cc:492] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:15.551433 7196 ts_tablet_manager.cc:1403] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.001s
I20260620 19:58:15.554309 7196 raft_consensus.cc:359] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.555220 7196 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:15.555876 7196 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.558997 7196 ts_tablet_manager.cc:1434] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:58:15.563500 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8cf0b0e2dafa4512b82ecefda3013d79 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:15.563903 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8cf0b0e2dafa4512b82ecefda3013d79 on tablet 8cf0b0e2dafa4512b82ecefda3013d79 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:15.625149 7196 ts_tablet_manager.cc:933] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:15.626823 7196 tablet_copy_client.cc:323] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:15.628357 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 1e39bd0f2de444d2acf683f9f18716da from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:15.628804 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 1e39bd0f2de444d2acf683f9f18716da from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-1e39bd0f2de444d2acf683f9f18716da
I20260620 19:58:15.634660 6391 tablet_copy_source_session.cc:215] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:15.637413 7196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e39bd0f2de444d2acf683f9f18716da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:15.650194 7196 tablet_copy_client.cc:806] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:15.650805 7196 tablet_copy_client.cc:670] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:15.654320 7196 tablet_copy_client.cc:538] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:15.660974 7196 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:15.687520 7196 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:15.688304 7196 tablet_bootstrap.cc:492] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:15.688925 7196 ts_tablet_manager.cc:1403] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.010s
I20260620 19:58:15.691565 7196 raft_consensus.cc:359] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.692145 7196 raft_consensus.cc:740] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:15.692656 7196 consensus_queue.cc:260] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:15.694713 7196 ts_tablet_manager.cc:1434] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 19:58:15.696277 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-1e39bd0f2de444d2acf683f9f18716da received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:15.696677 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-1e39bd0f2de444d2acf683f9f18716da on tablet 1e39bd0f2de444d2acf683f9f18716da with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:15.870141 6909 raft_consensus.cc:1217] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:15.880273 6909 raft_consensus.cc:1217] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260620 19:58:16.039733 6909 raft_consensus.cc:1217] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:16.222149 6909 raft_consensus.cc:1217] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:16.462183 7065 raft_consensus.cc:1064] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:16.464530 7065 consensus_queue.cc:237] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:16.475720 7148 raft_consensus.cc:1064] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:16.477926 7148 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:16.492025 6374 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:16.496411 6909 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:16.498411 7203 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:16.519755 6225 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:16.525343 7065 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:16.529676 6908 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.534718 7205 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.536173 6301 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.547195 6227 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.550837 7148 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.551689 7102 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260620 19:58:16.566334 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 74981.6us
I20260620 19:58:16.580121 7154 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.630947 7205 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.631785 7210 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.640511 7216 raft_consensus.cc:1064] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:16.643195 7216 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:16.644959 6908 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.657902 6129 catalog_manager.cc:5696] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:16.659951 6374 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.663024 6378 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.665081 7207 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.665917 6909 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.669390 6224 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.673349 7206 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.702736 6128 catalog_manager.cc:5696] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:16.706763 7148 raft_consensus.cc:1064] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:16.709553 7148 consensus_queue.cc:237] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:16.711630 6301 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.713058 6909 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.715530 6224 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.717903 6301 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.724900 7206 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.734814 6909 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.737910 7148 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.746292 6301 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.750214 7102 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.751245 6909 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.770696 6378 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.772748 7216 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:16.789817 6231 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:16.799039 7148 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:16.855521 6299 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:58:16.857539 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 44124.4us
W20260620 19:58:16.862717 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 60263.4us
W20260620 19:58:16.869890 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 62142.6us
I20260620 19:58:16.876876 6300 consensus_queue.cc:237] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:16.884513 6909 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:16.886191 6378 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:16.886500 7218 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:16.887775 7154 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:16.888913 6128 catalog_manager.cc:5696] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.897240 7154 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.899291 6378 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.905895 6910 raft_consensus.cc:2955] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.913208 7173 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.919796 6227 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.921891 6307 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.942978 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet cc7845798261439d855e526604de104a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:16.946048 6910 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.948726 6129 catalog_manager.cc:5696] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.969985 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet cc7845798261439d855e526604de104a with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:16.970794 6374 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:16.979562 6910 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:16.980604 6227 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:16.981824 7148 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:16.984583 7237 tablet_replica.cc:333] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:16.985575 7237 raft_consensus.cc:2243] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:16.986200 7237 raft_consensus.cc:2272] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:16.986739 7148 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:16.991449 6127 catalog_manager.cc:5696] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.992961 7231 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.994238 6910 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:16.996862 7237 ts_tablet_manager.cc:1916] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:17.009407 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 8dba7c3eac4b4392830be760b4060cbc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:17.015203 6227 raft_consensus.cc:2955] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.022965 6128 catalog_manager.cc:5696] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.028177 7237 ts_tablet_manager.cc:1929] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260620 19:58:17.028638 7237 log.cc:1199] T cc7845798261439d855e526604de104a P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/cc7845798261439d855e526604de104a
I20260620 19:58:17.030290 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet cc7845798261439d855e526604de104a (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:17.041510 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 8dba7c3eac4b4392830be760b4060cbc with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:17.044276 6378 consensus_queue.cc:237] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:17.044869 7238 tablet_replica.cc:333] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:17.048828 7238 raft_consensus.cc:2243] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:17.049580 7238 raft_consensus.cc:2272] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:17.053733 7238 ts_tablet_manager.cc:1916] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:17.054152 6910 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:17.054579 6227 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:17.055794 7148 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:17.056607 7205 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:17.064595 7148 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.070420 7238 ts_tablet_manager.cc:1929] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:17.070927 7238 log.cc:1199] T 8dba7c3eac4b4392830be760b4060cbc P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/8dba7c3eac4b4392830be760b4060cbc
I20260620 19:58:17.072570 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 8dba7c3eac4b4392830be760b4060cbc (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:17.072062 6227 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.074368 6910 raft_consensus.cc:2955] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.076984 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 1e39bd0f2de444d2acf683f9f18716da with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:17.090739 6128 catalog_manager.cc:5696] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.100108 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 1e39bd0f2de444d2acf683f9f18716da with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:17.100965 7238 tablet_replica.cc:333] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:17.101763 7238 raft_consensus.cc:2243] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:17.102309 7238 raft_consensus.cc:2272] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:17.105072 7238 ts_tablet_manager.cc:1916] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:17.118186 7238 ts_tablet_manager.cc:1929] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:17.118520 7238 log.cc:1199] T 1e39bd0f2de444d2acf683f9f18716da P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/1e39bd0f2de444d2acf683f9f18716da
I20260620 19:58:17.119817 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 1e39bd0f2de444d2acf683f9f18716da (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:17.240783 6227 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:17.246474 6910 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:17.246573 6307 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:17.248011 7216 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:17.249421 7233 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:17.256162 7207 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.259418 6910 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.260970 6307 raft_consensus.cc:2955] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.278990 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 8cf0b0e2dafa4512b82ecefda3013d79 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:17.283593 6130 catalog_manager.cc:5696] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:17.297405 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 8cf0b0e2dafa4512b82ecefda3013d79 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:17.299305 7239 tablet_replica.cc:333] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:17.300071 7239 raft_consensus.cc:2243] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:17.300596 7239 raft_consensus.cc:2272] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:17.303190 7239 ts_tablet_manager.cc:1916] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:17.317636 7239 ts_tablet_manager.cc:1929] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:17.318042 7239 log.cc:1199] T 8cf0b0e2dafa4512b82ecefda3013d79 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/8cf0b0e2dafa4512b82ecefda3013d79
I20260620 19:58:17.319319 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 8cf0b0e2dafa4512b82ecefda3013d79 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:18.366323 6307 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.372583 6226 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:18.373093 6271 consensus_peers.cc:597] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c21aa29d76d48fc933f76007685dae1. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:18.374037 7203 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:18.374526 6378 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:18.375782 7241 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:18.385491 7203 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.389348 6226 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.389412 6378 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.407747 6300 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.418299 6129 catalog_manager.cc:5696] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:18.427418 6226 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:18.427963 6378 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:18.428800 7154 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:18.429797 6271 consensus_peers.cc:597] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a566dbaea0746f4bc76e04b5072e045. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:18.429827 7203 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:18.443622 7241 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.445230 6378 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.459241 6226 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.465113 6300 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.468276 6130 catalog_manager.cc:5696] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:18.495289 6271 consensus_peers.cc:597] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d08b9c2da1448139f24e1d94c9616f3. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:18.495371 6378 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:18.496869 6226 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:18.496764 7241 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:18.498186 7154 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:18.505703 7245 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.507081 6226 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.513592 6378 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.520236 6227 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.524360 6130 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:18.536324 6196 consensus_peers.cc:597] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5739ffa3b914095862145adf0c084fb. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:18.540099 6378 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:18.541611 7206 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:18.550499 6300 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:18.559443 7226 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:18.569919 7206 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.570931 6378 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.572791 6300 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:18.585969 6130 catalog_manager.cc:5696] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:18.811134 7260 ts_tablet_manager.cc:933] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:18.812942 7260 tablet_copy_client.cc:323] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:18.814453 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 7a566dbaea0746f4bc76e04b5072e045 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:18.814988 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 7a566dbaea0746f4bc76e04b5072e045 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-7a566dbaea0746f4bc76e04b5072e045
I20260620 19:58:18.822386 6317 tablet_copy_source_session.cc:215] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:18.825891 7260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a566dbaea0746f4bc76e04b5072e045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:18.843619 7260 tablet_copy_client.cc:806] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:18.844197 7260 tablet_copy_client.cc:670] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:18.852442 7260 tablet_copy_client.cc:538] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:18.861222 7260 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:18.881696 7260 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:18.882710 7260 tablet_bootstrap.cc:492] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:18.883343 7260 ts_tablet_manager.cc:1403] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20260620 19:58:18.886274 7260 raft_consensus.cc:359] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.887125 7260 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:18.887315 7263 ts_tablet_manager.cc:933] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:18.887789 7260 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.889418 7263 tablet_copy_client.cc:323] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:18.891290 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 6d08b9c2da1448139f24e1d94c9616f3 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:18.891765 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 6d08b9c2da1448139f24e1d94c9616f3 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-6d08b9c2da1448139f24e1d94c9616f3
I20260620 19:58:18.892259 7260 ts_tablet_manager.cc:1434] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260620 19:58:18.897759 6316 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-7a566dbaea0746f4bc76e04b5072e045 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:18.898154 6316 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-7a566dbaea0746f4bc76e04b5072e045 on tablet 7a566dbaea0746f4bc76e04b5072e045 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:18.900782 6317 tablet_copy_source_session.cc:215] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:18.904915 7263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d08b9c2da1448139f24e1d94c9616f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:18.918198 7263 tablet_copy_client.cc:806] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:18.918727 7263 tablet_copy_client.cc:670] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:18.921952 7263 tablet_copy_client.cc:538] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:18.928614 7263 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:18.941943 7260 ts_tablet_manager.cc:933] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:18.947566 7260 tablet_copy_client.cc:323] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:18.948005 7263 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:18.948873 7263 tablet_bootstrap.cc:492] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:18.949494 7263 ts_tablet_manager.cc:1403] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20260620 19:58:18.950105 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 5c21aa29d76d48fc933f76007685dae1 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:18.950675 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 5c21aa29d76d48fc933f76007685dae1 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-5c21aa29d76d48fc933f76007685dae1
I20260620 19:58:18.959313 7263 raft_consensus.cc:359] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.960582 7263 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:18.961304 7263 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:18.962967 6317 tablet_copy_source_session.cc:215] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:18.966789 7263 ts_tablet_manager.cc:1434] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.017s user 0.007s sys 0.003s
I20260620 19:58:18.967263 7260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c21aa29d76d48fc933f76007685dae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:18.970885 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-6d08b9c2da1448139f24e1d94c9616f3 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:18.971297 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-6d08b9c2da1448139f24e1d94c9616f3 on tablet 6d08b9c2da1448139f24e1d94c9616f3 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:18.985219 7260 tablet_copy_client.cc:806] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:18.985798 7260 tablet_copy_client.cc:670] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:18.989349 7260 tablet_copy_client.cc:538] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:18.996898 7260 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:19.020079 7260 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:19.020795 7260 tablet_bootstrap.cc:492] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:19.021323 7260 ts_tablet_manager.cc:1403] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.005s
I20260620 19:58:19.023509 7260 raft_consensus.cc:359] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:19.024117 7260 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:19.024691 7260 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:19.028666 7260 ts_tablet_manager.cc:1434] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:58:19.030128 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-5c21aa29d76d48fc933f76007685dae1 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:19.030537 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-5c21aa29d76d48fc933f76007685dae1 on tablet 5c21aa29d76d48fc933f76007685dae1 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:19.132066 7260 ts_tablet_manager.cc:933] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:19.135733 7260 tablet_copy_client.cc:323] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:19.137176 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet a5739ffa3b914095862145adf0c084fb from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:19.137703 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet a5739ffa3b914095862145adf0c084fb from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-a5739ffa3b914095862145adf0c084fb
I20260620 19:58:19.144941 6242 tablet_copy_source_session.cc:215] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:19.147691 7260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5739ffa3b914095862145adf0c084fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:19.160326 7260 tablet_copy_client.cc:806] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:19.160902 7260 tablet_copy_client.cc:670] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:19.164208 7260 tablet_copy_client.cc:538] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:19.171272 7260 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:19.187898 7260 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:19.188661 7260 tablet_bootstrap.cc:492] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:19.189206 7260 ts_tablet_manager.cc:1403] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260620 19:58:19.191612 7260 raft_consensus.cc:359] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:19.192353 7260 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:19.192924 7260 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:19.194890 7260 ts_tablet_manager.cc:1434] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 19:58:19.198797 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a5739ffa3b914095862145adf0c084fb received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:19.199191 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a5739ffa3b914095862145adf0c084fb on tablet a5739ffa3b914095862145adf0c084fb with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:19.317771 6909 raft_consensus.cc:1217] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:19.441797 6909 raft_consensus.cc:1217] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:19.518277 6909 raft_consensus.cc:1217] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:19.627722 6909 raft_consensus.cc:1217] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:19.771239 7271 raft_consensus.cc:1064] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:19.773284 7271 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:19.779844 6378 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:19.781270 6227 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:19.782862 6909 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:19.783689 7272 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:19.785555 7272 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:19.787227 7245 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:19.799093 7245 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.800800 6909 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.811687 6374 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.808610 6227 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.831792 7272 raft_consensus.cc:1064] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:19.832072 6130 catalog_manager.cc:5696] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.834278 7272 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:19.876770 6374 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:19.878713 6909 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:19.878563 7245 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:19.881578 7154 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:19.884930 6227 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:19.890226 7245 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:19.907886 6299 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:19.926043 7154 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.930353 6910 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.940779 6374 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:19.939538 6227 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.945056 7271 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:19.948786 6374 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.953048 6909 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:19.958827 6128 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.962381 7272 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:19.964486 7154 raft_consensus.cc:1064] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:19.973659 7154 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:19.976107 6378 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.988278 7245 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:19.994234 6374 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:19.997169 7290 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:19.997234 6299 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:20.000330 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 7a566dbaea0746f4bc76e04b5072e045 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260620 19:58:20.006081 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 44815.1us
I20260620 19:58:20.009316 6226 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:20.009913 6908 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:20.012658 7245 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:20.019599 7272 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:20.023734 6909 raft_consensus.cc:2955] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.023917 6129 catalog_manager.cc:5696] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.034335 6910 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:20.035137 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a566dbaea0746f4bc76e04b5072e045 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:20.037175 7272 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:20.036031 6378 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:20.040434 7288 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:20.056787 7294 tablet_replica.cc:333] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:20.058714 7290 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.062227 7154 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.063026 6374 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.069099 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 6d08b9c2da1448139f24e1d94c9616f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:20.080406 6909 raft_consensus.cc:2955] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.083422 6227 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.085337 6128 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.085685 6374 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.057971 7294 raft_consensus.cc:2243] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:20.093072 7294 raft_consensus.cc:2272] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:20.099133 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet 6d08b9c2da1448139f24e1d94c9616f3 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:20.104779 7294 ts_tablet_manager.cc:1916] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:20.118589 6128 catalog_manager.cc:5696] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.132330 7294 ts_tablet_manager.cc:1929] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:20.132876 7294 log.cc:1199] T 7a566dbaea0746f4bc76e04b5072e045 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/7a566dbaea0746f4bc76e04b5072e045
I20260620 19:58:20.135365 7294 tablet_replica.cc:333] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:20.137101 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet 7a566dbaea0746f4bc76e04b5072e045 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:20.143956 6909 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.152463 7294 raft_consensus.cc:2243] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:20.156772 6299 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:20.162374 7294 raft_consensus.cc:2272] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:20.168318 7294 ts_tablet_manager.cc:1916] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:20.169597 6910 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:20.171025 6226 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:20.176079 7154 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:20.178141 7295 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:20.190521 7272 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.192314 6227 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.201193 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 5c21aa29d76d48fc933f76007685dae1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:20.203389 6910 raft_consensus.cc:2955] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.215471 6130 catalog_manager.cc:5696] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:20.217516 7294 ts_tablet_manager.cc:1929] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:20.218060 7294 log.cc:1199] T 6d08b9c2da1448139f24e1d94c9616f3 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/6d08b9c2da1448139f24e1d94c9616f3
I20260620 19:58:20.222800 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet 6d08b9c2da1448139f24e1d94c9616f3 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:20.225607 7206 raft_consensus.cc:1064] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:20.228317 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 5c21aa29d76d48fc933f76007685dae1 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:20.227834 7206 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:20.241478 7296 tablet_replica.cc:333] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:20.243328 7296 raft_consensus.cc:2243] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:20.243921 7296 raft_consensus.cc:2272] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:20.249969 7296 ts_tablet_manager.cc:1916] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:20.248762 6374 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:20.251832 6910 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:20.254158 6299 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:20.253198 7206 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260620 19:58:20.256613 7273 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:20.257409 7226 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:20.266299 7296 ts_tablet_manager.cc:1929] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:20.266911 7296 log.cc:1199] T 5c21aa29d76d48fc933f76007685dae1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/5c21aa29d76d48fc933f76007685dae1
I20260620 19:58:20.270197 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 5c21aa29d76d48fc933f76007685dae1 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:20.273811 7226 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.279969 6299 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.283058 6910 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.289116 6374 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.297638 6129 catalog_manager.cc:5696] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.317313 6227 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:20.327044 6910 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:20.328365 7206 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:20.332003 6374 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:20.333621 7206 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:20.336074 7226 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.345723 6378 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.348642 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet a5739ffa3b914095862145adf0c084fb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:20.360522 6910 raft_consensus.cc:2955] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:20.365149 6128 catalog_manager.cc:5696] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:20.375185 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet a5739ffa3b914095862145adf0c084fb with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:20.377180 7302 tablet_replica.cc:333] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:20.377918 7302 raft_consensus.cc:2243] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:20.378451 7302 raft_consensus.cc:2272] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:20.382323 7302 ts_tablet_manager.cc:1916] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:20.396054 7302 ts_tablet_manager.cc:1929] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:20.396425 7302 log.cc:1199] T a5739ffa3b914095862145adf0c084fb P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/a5739ffa3b914095862145adf0c084fb
I20260620 19:58:20.397636 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet a5739ffa3b914095862145adf0c084fb (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:21.443122 6226 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:21.453297 6196 consensus_peers.cc:597] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35398db5947a409b98b0d62a257ece06. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:21.454350 6299 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.454592 6378 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.455757 7206 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:21.456593 7301 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:21.465613 7305 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.467286 6299 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.504492 6128 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.513289 6299 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:21.515609 6378 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.537683 6378 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.539179 7154 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:21.539954 6271 consensus_peers.cc:597] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e56dc56fbc548ac997024a1b2e2ea1a. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:21.544236 6227 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.545290 7272 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:21.554253 7304 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.557039 6378 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.560148 6227 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.568737 6375 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:21.573071 6127 catalog_manager.cc:5696] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:21.594931 6299 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.596421 7317 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:21.597229 6345 consensus_peers.cc:597] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfeb1fb4ea6746dbaf0509f4097e00ab. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:21.597729 6227 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.599572 7275 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:21.605408 7275 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.606963 6299 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.608090 6224 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.620049 6375 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:21.623560 6127 catalog_manager.cc:5696] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.627950 6299 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.629376 7231 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:21.630059 6345 consensus_peers.cc:597] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65e57f1c5f9f437db061394645beec68. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:21.632362 6224 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:21.638916 7317 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:21.644721 7313 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.649915 6224 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.652009 6299 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:21.668625 6130 catalog_manager.cc:5696] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:21.945427 7325 ts_tablet_manager.cc:933] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:21.951665 7325 tablet_copy_client.cc:323] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:21.953929 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 35398db5947a409b98b0d62a257ece06 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:21.954686 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 35398db5947a409b98b0d62a257ece06 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-35398db5947a409b98b0d62a257ece06
I20260620 19:58:21.966266 6242 tablet_copy_source_session.cc:215] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:21.975378 7325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35398db5947a409b98b0d62a257ece06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:21.995114 7325 tablet_copy_client.cc:806] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:21.995754 7325 tablet_copy_client.cc:670] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:22.002178 7325 tablet_copy_client.cc:538] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:22.011608 7325 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:22.035589 7327 ts_tablet_manager.cc:933] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:22.070902 7327 tablet_copy_client.cc:323] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:22.073760 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 2e56dc56fbc548ac997024a1b2e2ea1a from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:22.074401 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 2e56dc56fbc548ac997024a1b2e2ea1a from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-2e56dc56fbc548ac997024a1b2e2ea1a
I20260620 19:58:22.095592 7325 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:22.096477 7325 tablet_bootstrap.cc:492] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:22.097203 7325 ts_tablet_manager.cc:1403] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.086s user 0.017s sys 0.009s
I20260620 19:58:22.100888 6317 tablet_copy_source_session.cc:215] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:22.101135 7325 raft_consensus.cc:359] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.104372 7325 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:22.104795 7327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e56dc56fbc548ac997024a1b2e2ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:22.105048 7325 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.114235 7325 ts_tablet_manager.cc:1434] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.017s user 0.004s sys 0.007s
I20260620 19:58:22.121893 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-35398db5947a409b98b0d62a257ece06 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:22.122457 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-35398db5947a409b98b0d62a257ece06 on tablet 35398db5947a409b98b0d62a257ece06 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:22.136148 7325 ts_tablet_manager.cc:933] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:22.150955 7325 tablet_copy_client.cc:323] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:22.151360 7327 tablet_copy_client.cc:806] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:22.151916 7327 tablet_copy_client.cc:670] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:22.153087 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 65e57f1c5f9f437db061394645beec68 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:22.153774 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 65e57f1c5f9f437db061394645beec68 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-65e57f1c5f9f437db061394645beec68
I20260620 19:58:22.156211 7327 tablet_copy_client.cc:538] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:22.159706 7331 ts_tablet_manager.cc:933] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:22.165607 6391 tablet_copy_source_session.cc:215] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:22.167208 7327 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:22.168434 7331 tablet_copy_client.cc:323] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:22.170149 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet cfeb1fb4ea6746dbaf0509f4097e00ab from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:22.170344 7325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e57f1c5f9f437db061394645beec68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:22.170858 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet cfeb1fb4ea6746dbaf0509f4097e00ab from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-cfeb1fb4ea6746dbaf0509f4097e00ab
I20260620 19:58:22.196175 6391 tablet_copy_source_session.cc:215] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:22.199612 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfeb1fb4ea6746dbaf0509f4097e00ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:22.202329 7325 tablet_copy_client.cc:806] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:22.203140 7325 tablet_copy_client.cc:670] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:22.208544 7325 tablet_copy_client.cc:538] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:22.221990 7327 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:22.222995 7327 tablet_bootstrap.cc:492] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:22.223613 7325 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:22.223634 7327 ts_tablet_manager.cc:1403] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.020s
I20260620 19:58:22.226927 7327 raft_consensus.cc:359] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.227720 7327 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:22.228374 7327 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.230901 7327 ts_tablet_manager.cc:1434] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 19:58:22.232458 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-2e56dc56fbc548ac997024a1b2e2ea1a received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:22.232892 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-2e56dc56fbc548ac997024a1b2e2ea1a on tablet 2e56dc56fbc548ac997024a1b2e2ea1a with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:22.238320 7331 tablet_copy_client.cc:806] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:22.238906 7331 tablet_copy_client.cc:670] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:22.246289 7331 tablet_copy_client.cc:538] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:22.255895 7331 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:22.260344 7325 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:22.261221 7325 tablet_bootstrap.cc:492] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:22.261821 7325 ts_tablet_manager.cc:1403] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.012s
I20260620 19:58:22.264948 7325 raft_consensus.cc:359] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.265741 7325 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:22.266323 7325 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.269023 7325 ts_tablet_manager.cc:1434] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20260620 19:58:22.273422 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-65e57f1c5f9f437db061394645beec68 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:22.273854 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-65e57f1c5f9f437db061394645beec68 on tablet 65e57f1c5f9f437db061394645beec68 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:22.280581 7331 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:22.281582 7331 tablet_bootstrap.cc:492] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:22.282274 7331 ts_tablet_manager.cc:1403] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.006s
I20260620 19:58:22.287009 7331 raft_consensus.cc:359] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.287631 7331 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:22.288232 7331 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:22.290588 7331 ts_tablet_manager.cc:1434] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260620 19:58:22.293077 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-cfeb1fb4ea6746dbaf0509f4097e00ab received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:22.293471 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-cfeb1fb4ea6746dbaf0509f4097e00ab on tablet cfeb1fb4ea6746dbaf0509f4097e00ab with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:22.320254 6910 raft_consensus.cc:1217] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:22.509958 6908 raft_consensus.cc:1217] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:22.554076 6908 raft_consensus.cc:1217] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:22.637043 6908 raft_consensus.cc:1217] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:22.878844 7206 raft_consensus.cc:1064] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:22.881368 7206 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:22.903707 6910 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:22.905540 6375 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:22.906648 7301 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:22.908716 6307 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:22.911298 7301 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:22.915122 7301 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:22.933207 7301 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:22.937687 6307 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:22.949685 6375 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:22.949877 6910 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:22.956254 6128 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:22.989668 6227 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:22.997965 6910 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:22.998648 6307 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:23.000473 7305 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.002295 7206 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.007472 7305 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.009773 6908 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.016028 6307 raft_consensus.cc:2955] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.019833 7154 raft_consensus.cc:1064] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:23.024459 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 35398db5947a409b98b0d62a257ece06 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:23.022200 7154 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:23.033685 7317 raft_consensus.cc:1064] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:23.042050 7317 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:23.044703 6127 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.059659 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 35398db5947a409b98b0d62a257ece06 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:23.080777 6378 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.081038 7350 tablet_replica.cc:333] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:23.082108 7350 raft_consensus.cc:2243] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:23.083269 7350 raft_consensus.cc:2272] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:23.082969 7304 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:23.083485 6908 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.085927 7304 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:23.090545 6227 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.093362 6910 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.094755 7350 ts_tablet_manager.cc:1916] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:23.094923 7231 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:23.100497 6224 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.102270 7231 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:23.103736 6301 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.106199 7313 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:23.093348 7304 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260620 19:58:23.121654 7350 ts_tablet_manager.cc:1929] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:23.122256 7350 log.cc:1199] T 35398db5947a409b98b0d62a257ece06 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/35398db5947a409b98b0d62a257ece06
I20260620 19:58:23.124048 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 35398db5947a409b98b0d62a257ece06 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:23.135048 7313 raft_consensus.cc:1064] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:23.137483 7313 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:23.142975 7317 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.150434 6910 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.170800 6307 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.175191 7231 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:23.176453 7154 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.176920 6129 catalog_manager.cc:5696] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.185974 6301 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.193179 6378 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.195099 6908 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.225602 6909 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.228370 6226 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:23.235334 6226 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.239411 7231 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:23.243345 7360 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:23.249187 6224 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.289100 6375 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:23.290266 6130 catalog_manager.cc:5696] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.303864 7356 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.313877 6307 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:23.321638 6910 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:23.322466 6226 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:23.324406 7313 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.323799 6224 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.325835 6908 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:23.327719 6227 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:23.329331 7154 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.340986 7337 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.353168 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 2e56dc56fbc548ac997024a1b2e2ea1a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:23.352458 6908 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.360994 6127 catalog_manager.cc:5696] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:23.365645 6129 catalog_manager.cc:5696] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:23.372291 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e56dc56fbc548ac997024a1b2e2ea1a with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:23.373232 7350 tablet_replica.cc:333] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:23.374230 7350 raft_consensus.cc:2243] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:23.375046 7350 raft_consensus.cc:2272] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:23.380277 7360 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.381857 7313 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.386757 7350 ts_tablet_manager.cc:1916] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:23.392475 6911 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.397192 6375 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:23.407131 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 65e57f1c5f9f437db061394645beec68 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260620 19:58:23.410065 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 69694.4us
I20260620 19:58:23.412393 7154 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.414263 6911 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.417368 6129 catalog_manager.cc:5696] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.419914 6301 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.433492 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 65e57f1c5f9f437db061394645beec68 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:23.443590 7350 ts_tablet_manager.cc:1929] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:23.444100 7350 log.cc:1199] T 2e56dc56fbc548ac997024a1b2e2ea1a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/2e56dc56fbc548ac997024a1b2e2ea1a
I20260620 19:58:23.447608 6300 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:23.449693 6227 raft_consensus.cc:2955] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.451223 6908 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:23.453835 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 2e56dc56fbc548ac997024a1b2e2ea1a (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:23.454939 6224 raft_consensus.cc:2955] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.465847 7365 tablet_replica.cc:333] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:23.470345 7317 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.470345 7365 raft_consensus.cc:2243] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:23.474196 7365 raft_consensus.cc:2272] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:23.481964 7365 ts_tablet_manager.cc:1916] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:23.483749 7360 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:23.490023 6300 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.488518 7360 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.508652 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet cfeb1fb4ea6746dbaf0509f4097e00ab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:23.524051 7365 ts_tablet_manager.cc:1929] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:23.523394 6130 catalog_manager.cc:5696] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.524521 7365 log.cc:1199] T 65e57f1c5f9f437db061394645beec68 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/65e57f1c5f9f437db061394645beec68
I20260620 19:58:23.526232 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 65e57f1c5f9f437db061394645beec68 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:23.531987 6911 raft_consensus.cc:2955] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:23.536550 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet cfeb1fb4ea6746dbaf0509f4097e00ab with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:23.540432 7366 tablet_replica.cc:333] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:23.541774 7366 raft_consensus.cc:2243] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:23.542591 7366 raft_consensus.cc:2272] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:23.547322 7366 ts_tablet_manager.cc:1916] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:23.563084 7366 ts_tablet_manager.cc:1929] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:23.563541 7366 log.cc:1199] T cfeb1fb4ea6746dbaf0509f4097e00ab P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/cfeb1fb4ea6746dbaf0509f4097e00ab
I20260620 19:58:23.565286 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet cfeb1fb4ea6746dbaf0509f4097e00ab (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:23.859423 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:23.908859 6300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:23.909528 6300 raft_consensus.cc:606] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:23.923570 6375 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79a07a8804784405920baab70a535338"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:23.924022 6375 raft_consensus.cc:606] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:23.926128 6300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:23.926577 6300 raft_consensus.cc:606] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:23.928627 6300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:23.928978 6300 raft_consensus.cc:606] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:23.931049 6300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a56f959cee394e398810b9170b9d24be"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:23.931406 6300 raft_consensus.cc:606] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:23.933421 6300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:23.933827 6300 raft_consensus.cc:606] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:23.934728 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260620 19:58:23.935917 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:23.936870 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:24.069152 7154 raft_consensus.cc:993] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.069655 7337 raft_consensus.cc:1081] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.070988 6227 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:24.071382 6227 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:24.071604 6227 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.075563 6227 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.076804 6227 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:24.077517 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:24.077684 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:24.078192 6375 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.078161 6301 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:24.078715 6301 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:24.079453 6301 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.080590 6301 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.084892 6375 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:24.086163 6198 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:24.087091 7349 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:24.087349 6301 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:24.087491 7349 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:24.088389 7349 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.096838 6130 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:24.231245 7154 raft_consensus.cc:993] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.237550 7375 raft_consensus.cc:993] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.237797 7376 raft_consensus.cc:1081] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.238081 7364 raft_consensus.cc:1081] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.239385 6227 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a56f959cee394e398810b9170b9d24be"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:24.239892 6227 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:24.240257 6227 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:24.240068 6224 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:24.240686 6224 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:24.241055 6224 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.248188 6227 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.250582 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:24.251096 6301 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:24.251379 6301 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:24.252099 6301 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.252612 6224 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.253124 6301 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:24.254375 6224 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:24.255367 6227 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:24.255270 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:24.255889 6375 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.256721 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:24.257241 6374 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:24.260540 6301 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:24.261587 6195 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:24.262291 7348 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:24.262774 7348 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:24.263258 6375 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:24.263541 6374 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:24.263621 7348 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.265026 6198 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:24.266273 7349 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:24.266765 7349 raft_consensus.cc:697] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:24.267795 7349 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.272392 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:24.273042 6301 raft_consensus.cc:3055] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:24.273422 6301 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:24.274104 6301 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.279368 6301 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.290311 6130 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:24.296648 6301 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:24.298379 7375 raft_consensus.cc:993] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.298830 7375 raft_consensus.cc:1081] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:24.301281 6224 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:24.301771 6224 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:24.302084 6224 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.309341 7337 raft_consensus.cc:993] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:24.309856 7337 raft_consensus.cc:1081] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:24.311378 6911 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53408
I20260620 19:58:24.311925 6911 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:24.311561 6224 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.312264 6911 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.314427 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:24.314999 6224 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:24.315940 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:24.316465 6374 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.318727 6130 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:24.314965 6300 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:24.320763 6300 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:24.321527 6300 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.322914 6300 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.323021 6374 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:24.324357 6198 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:24.325636 7348 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:24.326099 7348 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:24.326974 7348 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.332646 6300 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:24.336027 6911 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:24.338279 6911 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:24.339180 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:24.339835 6374 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.339826 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:24.340533 6300 raft_consensus.cc:3055] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:24.341500 6300 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:24.342190 6300 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:24.343214 6300 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.343159 6130 catalog_manager.cc:5696] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:24.346732 6374 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:24.347966 6877 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:24.349380 6300 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:24.350237 7382 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:24.358843 7382 raft_consensus.cc:697] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:24.359508 7382 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:24.367204 6130 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:24.441459 7356 raft_consensus.cc:993] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:24.441978 7317 raft_consensus.cc:1081] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:24.443639 6301 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "79a07a8804784405920baab70a535338"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:24.444139 6301 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:24.444396 6301 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.448891 6301 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.450354 6301 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:24.450965 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:24.451486 6224 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.451242 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:24.452210 6374 raft_consensus.cc:3055] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:24.452495 6374 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:24.453290 6374 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.454258 6374 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:24.456410 6224 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:24.458010 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:24.459120 7375 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:24.459502 6374 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:24.459560 7375 raft_consensus.cc:697] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:24.461894 7375 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:24.474123 6130 catalog_manager.cc:5696] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:24.605043 6227 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:24.610625 6374 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:24.610924 6196 consensus_peers.cc:597] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb61840b191946f9a272fb1b78f28cec. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:24.611135 6301 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:24.612200 7348 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.613029 7349 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.621070 7348 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.623638 6301 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.623241 6374 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.635078 6300 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:24.640084 6130 catalog_manager.cc:5696] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.645589 6374 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:24.646675 6227 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:24.646919 7375 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:24.646019 6271 consensus_peers.cc:597] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddb66400acd140a5b26f1a19f2abd18e. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:24.648515 7374 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.655169 7337 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.656942 6374 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.671169 6227 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.672257 6224 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:24.672708 6130 catalog_manager.cc:5696] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.677292 6374 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:24.678761 7348 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260620 19:58:24.690799 6196 consensus_peers.cc:597] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0153cc2373244ab0b2574d1de595c4a4. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:24.693079 6301 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:24.695766 6378 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:24.697978 6377 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:24.698796 7379 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.700191 7379 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.700968 7349 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.702069 6377 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:24.708956 7379 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:24.713322 6300 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:24.715287 7379 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.719584 6307 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:24.720053 6299 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:24.724473 7377 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.725996 7377 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.766163 7377 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.794065 6299 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.814888 6377 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:24.820886 6378 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.819979 6377 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:24.827881 7410 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.833668 6307 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:24.834023 6345 consensus_peers.cc:597] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03268245eb53439a9b88defe52240125. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:24.835587 7313 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.896791 6300 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:24.898267 6301 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:24.900213 6129 catalog_manager.cc:5696] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:24.902045 7416 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:24.904596 6301 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.903666 7313 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.929325 6227 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:24.942660 7356 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:24.950521 7411 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:24.960232 6129 catalog_manager.cc:5696] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:24.976336 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:24.982878 6224 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:25.007040 6377 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:25.026203 7418 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:25.047547 6224 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:25.049309 7375 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:25.071693 6377 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:25.074033 7337 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:25.109998 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:25.110566 6307 raft_consensus.cc:606] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.128535 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:25.129097 6307 raft_consensus.cc:606] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.131780 6377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "64d372664b48432eafee6464408fcef5"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:25.133042 6377 raft_consensus.cc:606] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:25.133785 7426 ts_tablet_manager.cc:933] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:25.140090 6378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:25.140642 6378 raft_consensus.cc:606] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.143072 7426 tablet_copy_client.cc:323] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:25.145404 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet ddb66400acd140a5b26f1a19f2abd18e from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:25.145900 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet ddb66400acd140a5b26f1a19f2abd18e from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-ddb66400acd140a5b26f1a19f2abd18e
I20260620 19:58:25.161206 6317 tablet_copy_source_session.cc:215] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:25.164523 7426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb66400acd140a5b26f1a19f2abd18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:25.171268 6224 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a56f959cee394e398810b9170b9d24be"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:25.171886 6224 raft_consensus.cc:606] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:25.174646 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:25.175161 6307 raft_consensus.cc:606] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.176308 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260620 19:58:25.177731 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:25.178397 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:25.183780 7431 ts_tablet_manager.cc:933] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:25.186661 7426 tablet_copy_client.cc:806] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:25.187294 7426 tablet_copy_client.cc:670] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:25.189785 7431 tablet_copy_client.cc:323] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:25.191365 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet eb61840b191946f9a272fb1b78f28cec from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:25.191846 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet eb61840b191946f9a272fb1b78f28cec from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-eb61840b191946f9a272fb1b78f28cec
I20260620 19:58:25.197731 6242 tablet_copy_source_session.cc:215] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:25.198460 7426 tablet_copy_client.cc:538] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:25.201319 7431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb61840b191946f9a272fb1b78f28cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:25.212236 7426 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:25.222800 7431 tablet_copy_client.cc:806] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:25.232021 7431 tablet_copy_client.cc:670] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:25.251876 7337 raft_consensus.cc:993] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.253935 7337 raft_consensus.cc:1081] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.252676 7431 tablet_copy_client.cc:538] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:25.257952 6911 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53408
I20260620 19:58:25.258467 6911 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:25.258808 6911 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.261279 7426 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:25.262177 7426 tablet_bootstrap.cc:492] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:25.262817 7426 ts_tablet_manager.cc:1403] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.051s user 0.026s sys 0.017s
I20260620 19:58:25.264562 7431 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:25.265614 7426 raft_consensus.cc:359] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.266466 7426 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:25.267113 7426 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.269138 6911 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.272655 7426 ts_tablet_manager.cc:1434] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260620 19:58:25.274827 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:25.275487 6224 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.276098 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:25.276603 6307 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:25.276871 6307 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:25.277539 6307 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.278502 6307 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.279749 6911 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:25.281497 6224 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:25.282719 6875 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:25.283543 7418 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:25.284022 7418 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:25.284066 6307 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:25.285025 7418 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.285578 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-ddb66400acd140a5b26f1a19f2abd18e received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:25.286010 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-ddb66400acd140a5b26f1a19f2abd18e on tablet ddb66400acd140a5b26f1a19f2abd18e with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.292949 7426 ts_tablet_manager.cc:933] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:25.296707 7420 raft_consensus.cc:993] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:25.297180 7420 raft_consensus.cc:1081] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:25.299106 6299 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "64d372664b48432eafee6464408fcef5"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:25.299626 6299 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:25.299981 6299 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:25.317596 7431 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:25.318523 7431 tablet_bootstrap.cc:492] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:25.319195 7431 ts_tablet_manager.cc:1403] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.055s user 0.013s sys 0.013s
I20260620 19:58:25.322221 7431 raft_consensus.cc:359] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.323066 7317 raft_consensus.cc:993] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.323560 7317 raft_consensus.cc:1081] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.325052 6908 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53406
I20260620 19:58:25.325547 6908 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:25.325893 6908 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.339712 7437 ts_tablet_manager.cc:933] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:25.341141 6908 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.323066 7431 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:25.347576 7431 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.348964 6908 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:25.350414 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:25.350951 6307 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.352319 7431 ts_tablet_manager.cc:1434] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.033s user 0.005s sys 0.002s
I20260620 19:58:25.351842 6128 catalog_manager.cc:5696] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:25.354118 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-eb61840b191946f9a272fb1b78f28cec received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:25.354569 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-eb61840b191946f9a272fb1b78f28cec on tablet eb61840b191946f9a272fb1b78f28cec with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.355685 7437 tablet_copy_client.cc:323] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:25.357301 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 03268245eb53439a9b88defe52240125 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:25.357808 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 03268245eb53439a9b88defe52240125 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-03268245eb53439a9b88defe52240125
I20260620 19:58:25.366416 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:25.366777 6391 tablet_copy_source_session.cc:215] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:25.367072 6377 raft_consensus.cc:3055] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:25.367404 6377 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:25.368141 6377 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.369385 6377 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.375694 6307 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:25.377187 6874 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:25.377992 7418 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:25.378449 7418 raft_consensus.cc:697] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:25.379274 7418 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.382283 6377 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:25.384469 7337 raft_consensus.cc:993] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.385042 7337 raft_consensus.cc:1081] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.385269 7437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03268245eb53439a9b88defe52240125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:25.386685 6911 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53408
I20260620 19:58:25.387172 6911 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:25.387483 6911 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.387799 6299 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:25.393474 6911 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.397006 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:25.397527 6307 raft_consensus.cc:3055] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:25.397802 6307 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:25.398547 6307 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.399556 6307 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.402249 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:25.402815 6377 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.409438 6911 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:25.418479 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:25.419188 6226 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:25.421023 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:25.421576 6378 raft_consensus.cc:3055] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:25.421861 6378 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:25.422566 6378 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:25.423627 6378 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:25.423399 6128 catalog_manager.cc:5696] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:25.425066 6226 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:25.428573 6271 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:25.429456 7337 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:25.429916 7337 raft_consensus.cc:697] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:25.431494 7337 consensus_queue.cc:237] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:25.440510 6378 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:25.441809 6307 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:25.443007 6874 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:25.443960 7418 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:25.444417 7418 raft_consensus.cc:697] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:25.445200 7418 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.447973 6299 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:25.450510 7411 raft_consensus.cc:993] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:25.450932 7411 raft_consensus.cc:1081] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:25.451906 7426 tablet_copy_client.cc:323] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:25.454893 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 0153cc2373244ab0b2574d1de595c4a4 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:25.455405 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 0153cc2373244ab0b2574d1de595c4a4 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-0153cc2373244ab0b2574d1de595c4a4
I20260620 19:58:25.454274 6307 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a56f959cee394e398810b9170b9d24be"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52350
I20260620 19:58:25.458478 6307 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:25.460733 6307 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:25.466423 6130 catalog_manager.cc:5696] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:25.470255 6242 tablet_copy_source_session.cc:215] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:25.471359 6377 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:25.479156 6307 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:25.479081 6127 catalog_manager.cc:5696] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:25.481987 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:25.482514 6378 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:25.483901 7437 tablet_copy_client.cc:806] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:25.484426 7437 tablet_copy_client.cc:670] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:25.486763 7426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0153cc2373244ab0b2574d1de595c4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:25.493875 6378 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:25.495028 6307 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:25.495544 6273 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:25.496495 7337 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:25.496922 7337 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:25.497615 7337 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:25.499480 7437 tablet_copy_client.cc:538] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:25.506310 7426 tablet_copy_client.cc:806] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:25.506866 7426 tablet_copy_client.cc:670] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:25.508704 7437 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:25.508605 6127 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 4 to 5, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 5 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:25.515787 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:25.516379 6226 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:25.516682 6226 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:25.517377 6226 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:25.518512 6226 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:25.521281 7375 raft_consensus.cc:993] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.521733 7375 raft_consensus.cc:1081] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:25.524907 6226 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:25.533972 6908 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53408
I20260620 19:58:25.534703 6908 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:25.535120 6908 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.537317 7426 tablet_copy_client.cc:538] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:25.544524 6908 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.546236 7426 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:25.547683 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:25.548261 6300 raft_consensus.cc:3055] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:25.548535 6300 raft_consensus.cc:740] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:25.549185 6300 consensus_queue.cc:260] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.550218 6300 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.550788 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:25.551077 6908 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:25.551870 6377 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:25.558300 6300 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:25.559646 6874 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:25.560763 7419 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:25.561319 7419 raft_consensus.cc:697] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:25.562361 7419 consensus_queue.cc:237] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:25.567711 6377 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
W20260620 19:58:25.616070 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 51197.9us
I20260620 19:58:25.619438 6908 raft_consensus.cc:1217] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:25.628921 7437 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:25.634474 7437 tablet_bootstrap.cc:492] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:25.635260 7437 ts_tablet_manager.cc:1403] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.127s user 0.026s sys 0.036s
I20260620 19:58:25.635926 6127 catalog_manager.cc:5696] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:25.639132 7437 raft_consensus.cc:359] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.640022 7437 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:25.640734 7437 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.643559 7437 ts_tablet_manager.cc:1434] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:58:25.645143 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-03268245eb53439a9b88defe52240125 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:25.645561 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-03268245eb53439a9b88defe52240125 on tablet 03268245eb53439a9b88defe52240125 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.655648 7426 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:25.656601 7426 tablet_bootstrap.cc:492] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:25.657233 7426 ts_tablet_manager.cc:1403] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.111s user 0.024s sys 0.002s
I20260620 19:58:25.659981 7426 raft_consensus.cc:359] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.660775 7426 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:25.661375 7426 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:25.663863 7426 ts_tablet_manager.cc:1434] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:58:25.665421 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-0153cc2373244ab0b2574d1de595c4a4 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:25.665814 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-0153cc2373244ab0b2574d1de595c4a4 on tablet 0153cc2373244ab0b2574d1de595c4a4 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:25.698057 6307 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:25.699318 7418 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:25.714844 6224 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:25.716526 7446 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:25.724713 6911 raft_consensus.cc:1217] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:25.733961 6911 raft_consensus.cc:1217] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:25.759294 6908 raft_consensus.cc:1217] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:25.877795 6377 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:25.878912 7446 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:25.894270 6301 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:25.896824 7448 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:25.906919 6224 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:25.910377 7374 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:25.922101 6301 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:25.923663 7448 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:25.925395 6377 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:25.927754 7375 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:25.976099 6378 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:25.977679 7446 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:26.001893 6375 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260620 19:58:26.003285 7443 raft_consensus.cc:1064] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:26.007988 7337 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:26.005901 7443 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.017637 6226 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260620 19:58:26.019155 7337 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:26.033751 6378 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:26.049638 7447 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:26.064491 6908 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.065842 7375 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:26.069828 6377 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.072239 7464 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:26.083444 6226 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.086189 7443 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:26.107935 6307 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:26.115612 7448 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:26.163733 7337 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.167183 6375 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.175143 6908 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.178850 6226 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.186820 6128 catalog_manager.cc:5696] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.224960 6300 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.231499 6908 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:26.232194 6375 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:26.232704 7472 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:26.233561 7443 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:26.238411 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:26.244203 7443 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.248939 6375 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.255810 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet ddb66400acd140a5b26f1a19f2abd18e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:26.251567 6908 raft_consensus.cc:2955] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.265166 6127 catalog_manager.cc:5696] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:26.280083 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet ddb66400acd140a5b26f1a19f2abd18e with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:26.298442 7476 tablet_replica.cc:333] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:26.299551 7476 raft_consensus.cc:2243] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:26.300328 7476 raft_consensus.cc:2272] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:26.309180 7476 ts_tablet_manager.cc:1916] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:26.325601 7476 ts_tablet_manager.cc:1929] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:26.326063 7476 log.cc:1199] T ddb66400acd140a5b26f1a19f2abd18e P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/ddb66400acd140a5b26f1a19f2abd18e
I20260620 19:58:26.327706 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet ddb66400acd140a5b26f1a19f2abd18e (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:26.329916 7313 raft_consensus.cc:1064] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:26.332103 7313 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.333503 6375 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:26.334079 6375 raft_consensus.cc:606] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:26.337497 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "35398db5947a409b98b0d62a257ece06"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:26.338040 6231 raft_consensus.cc:606] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:26.339385 6908 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.340404 7411 raft_consensus.cc:1064] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:26.341216 6375 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92212144534e4394844003016ba67a47"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:26.341435 7317 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:26.341742 6375 raft_consensus.cc:606] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:26.343366 7411 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.344470 6300 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.345781 7420 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:26.346736 6375 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "99e667a051414b87955b78b2520409b5"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:26.347258 6375 raft_consensus.cc:606] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:26.349290 6231 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.352784 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:26.353291 6307 raft_consensus.cc:606] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:26.354862 7420 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:26.358067 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:26.358548 6307 raft_consensus.cc:606] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:26.359613 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260620 19:58:26.361136 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:26.363476 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:26.373670 7420 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.388845 6908 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.389910 6300 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.398885 6231 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.398999 6129 catalog_manager.cc:5696] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:26.405886 6911 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.420578 7484 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:26.436357 7348 raft_consensus.cc:1064] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:26.440886 7348 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.449424 7470 raft_consensus.cc:993] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:26.449936 7470 raft_consensus.cc:1081] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:26.452222 6307 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "99e667a051414b87955b78b2520409b5"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:26.452822 6307 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:26.453295 6307 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.481171 6307 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.485206 6307 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:26.488873 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:26.489435 6375 raft_consensus.cc:3055] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:26.489724 6375 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:26.490443 6375 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.491798 6375 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.493880 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:26.494429 6226 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.494638 6377 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.500880 6226 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:26.503659 6301 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.503949 6375 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:26.504292 6271 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:26.502472 7470 raft_consensus.cc:993] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:26.507754 7472 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:26.508230 7472 raft_consensus.cc:697] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:26.509043 7472 consensus_queue.cc:237] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.511540 7485 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260620 19:58:26.512728 7317 raft_consensus.cc:1081] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:26.514547 6307 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:26.515079 6307 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:26.515420 6307 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.513453 7486 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:26.582508 6375 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.587585 6911 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.607355 6300 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:26.611413 6129 catalog_manager.cc:5696] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:26.611874 7317 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:26.622032 6377 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.626368 7470 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.631349 6299 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:26.632095 6908 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:26.635846 7313 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:26.632189 6307 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.640503 6307 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:26.637084 7484 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260620 19:58:26.643006 6908 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.649127 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:26.649853 6377 raft_consensus.cc:3055] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:26.650182 6377 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:26.650931 6377 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.653749 6377 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.654037 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:26.654666 6231 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.656807 7407 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260620 19:58:26.653720 7490 log.cc:927] Time spent T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260620 19:58:26.667714 6377 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:26.670725 6273 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:26.671617 7464 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:26.672374 7464 raft_consensus.cc:697] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:26.673470 7464 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:26.680810 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 60720.7us
I20260620 19:58:26.680876 7487 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:26.691634 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 03268245eb53439a9b88defe52240125 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:26.700174 6231 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
W20260620 19:58:26.670347 7492 log.cc:927] Time spent T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.076s user 0.002s sys 0.000s
I20260620 19:58:26.729893 6307 raft_consensus.cc:2955] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.755086 6127 catalog_manager.cc:5696] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:26.764223 7472 raft_consensus.cc:993] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:26.774057 7443 raft_consensus.cc:1081] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:26.774732 7470 raft_consensus.cc:993] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:26.775226 7470 raft_consensus.cc:1081] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
W20260620 19:58:26.781282 7494 log.cc:927] Time spent T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260620 19:58:26.781657 7484 raft_consensus.cc:993] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:26.782099 7485 raft_consensus.cc:1081] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:26.783870 6910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "35398db5947a409b98b0d62a257ece06"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53396
I20260620 19:58:26.784373 6910 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:26.784700 6910 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.786139 6301 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92212144534e4394844003016ba67a47"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:26.786665 6301 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:26.786983 6301 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.791065 7485 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.798141 6307 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.803303 6130 catalog_manager.cc:5696] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:26.806831 7374 raft_consensus.cc:993] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:26.807334 7374 raft_consensus.cc:1081] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:26.790591 7493 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.792871 6911 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53408
I20260620 19:58:26.793617 6908 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.809499 6911 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:26.809871 6911 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.796607 6910 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.813843 6907 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53408
I20260620 19:58:26.814397 6907 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:26.814757 6907 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.813383 6301 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.826627 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet 03268245eb53439a9b88defe52240125 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:26.827934 7476 tablet_replica.cc:333] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:26.827842 6911 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.829402 7476 raft_consensus.cc:2243] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:26.830008 7476 raft_consensus.cc:2272] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:26.831341 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:26.831889 6307 raft_consensus.cc:3055] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:26.832187 6307 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:26.833076 6911 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:26.836568 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:26.837158 6224 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.837384 6910 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:26.839764 6307 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.840744 6301 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:26.840821 6307 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.844079 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:26.844640 6375 raft_consensus.cc:3055] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:26.844908 6375 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:26.842056 6129 catalog_manager.cc:5696] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.849956 6299 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.852334 7476 ts_tablet_manager.cc:1916] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:26.856655 6377 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.858093 6307 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:26.860898 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:26.861399 6226 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.856041 6378 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.822821 6907 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.862591 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:26.867252 6301 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.869983 6874 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:26.871207 6224 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:26.865861 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:26.873674 7444 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:26.874161 7444 raft_consensus.cc:697] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:26.875778 6301 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:26.875685 6375 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.875002 7444 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.887202 6375 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.899643 6227 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.904407 6375 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:26.906661 6874 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:26.914943 7446 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:26.915426 7446 raft_consensus.cc:697] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:26.916283 7446 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.892118 6231 raft_consensus.cc:3055] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:26.919349 6226 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:26.915922 6130 catalog_manager.cc:5696] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:26.920742 6273 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:26.925554 7337 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:26.926105 7337 raft_consensus.cc:697] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:26.932461 7337 consensus_queue.cc:237] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:26.922209 6231 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:26.951543 6907 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:26.952515 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:26.954686 6301 raft_consensus.cc:3055] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:26.955011 6301 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:26.955730 6301 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:26.956259 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:26.956744 6301 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.956825 6378 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:26.948909 6226 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:26.963786 7443 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:26.953920 6377 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:26.971026 6378 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:26.972635 7348 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:26.974896 6877 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:26.975638 7448 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:26.976251 7448 raft_consensus.cc:697] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:26.977041 7448 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.005096 6301 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:27.009204 6907 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:27.012295 7348 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:27.000764 6231 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.034986 6231 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:27.040160 7487 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.048183 6375 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.073400 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet eb61840b191946f9a272fb1b78f28cec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:27.090651 6910 raft_consensus.cc:2955] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.082136 6231 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
W20260620 19:58:27.103463 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 83079.9us
I20260620 19:58:27.099743 6130 catalog_manager.cc:5696] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:27.107506 7476 ts_tablet_manager.cc:1929] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:27.107911 7476 log.cc:1199] T 03268245eb53439a9b88defe52240125 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/03268245eb53439a9b88defe52240125
I20260620 19:58:27.108355 6378 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:27.109378 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet 03268245eb53439a9b88defe52240125 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:27.081933 6907 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.123857 6129 catalog_manager.cc:5696] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:27.131920 6231 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:27.133760 7443 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260620 19:58:27.148087 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 77130.4us
I20260620 19:58:27.147310 6127 catalog_manager.cc:5696] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:27.181034 7503 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260620 19:58:27.200810 6375 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:27.205401 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet eb61840b191946f9a272fb1b78f28cec with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:27.209347 7503 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:27.213184 6224 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.222409 6907 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:27.222970 6129 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:27.223989 7512 tablet_replica.cc:333] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:27.224578 6129 catalog_manager.cc:5696] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:27.226188 7486 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:27.241267 6301 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:27.240764 7411 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.245543 7505 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:27.248061 6910 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.266541 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 0153cc2373244ab0b2574d1de595c4a4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:27.266522 7512 raft_consensus.cc:2243] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:27.267741 7512 raft_consensus.cc:2272] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:27.271715 6299 raft_consensus.cc:2955] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.274070 6224 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:27.275520 7446 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260620 19:58:27.276661 6115 catalog_manager.cc:5219] ChangeConfig:REMOVE_PEER RPC for tablet 0153cc2373244ab0b2574d1de595c4a4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20260620 19:58:27.285229 6127 catalog_manager.cc:5696] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:27.290534 7512 ts_tablet_manager.cc:1916] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:27.308485 7512 ts_tablet_manager.cc:1929] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:27.309825 7512 log.cc:1199] T eb61840b191946f9a272fb1b78f28cec P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/eb61840b191946f9a272fb1b78f28cec
I20260620 19:58:27.313390 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 0153cc2373244ab0b2574d1de595c4a4 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:27.313659 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet eb61840b191946f9a272fb1b78f28cec (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:27.316504 7516 tablet_replica.cc:333] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:27.321710 7516 raft_consensus.cc:2243] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:27.322908 6299 raft_consensus.cc:1275] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:27.323347 7516 raft_consensus.cc:2272] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:27.326821 7444 consensus_queue.cc:1048] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:27.327191 7516 ts_tablet_manager.cc:1916] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:27.341403 7516 ts_tablet_manager.cc:1929] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:27.341765 7516 log.cc:1199] T 0153cc2373244ab0b2574d1de595c4a4 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/0153cc2373244ab0b2574d1de595c4a4
I20260620 19:58:27.343068 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 0153cc2373244ab0b2574d1de595c4a4 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:27.356660 6224 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:27.357784 7504 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:27.369058 6378 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:27.370323 7504 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:27.430537 6301 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:27.431612 7444 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:27.439731 6224 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:27.441181 7448 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:27.552196 6301 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:27.552645 6378 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:27.553552 7446 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:27.554253 7448 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:27.638561 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:27.673390 6301 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:27.673966 6301 raft_consensus.cc:606] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:27.676607 6224 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:27.677201 6224 raft_consensus.cc:606] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:27.679937 6299 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "99e667a051414b87955b78b2520409b5"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:27.680471 6299 raft_consensus.cc:606] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:27.683398 6378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:27.683939 6378 raft_consensus.cc:606] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:27.685279 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260620 19:58:27.686503 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:27.687232 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:27.803730 7337 raft_consensus.cc:993] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:27.804350 7472 raft_consensus.cc:1081] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:27.805783 6910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53408
I20260620 19:58:27.806310 6910 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:27.806670 6910 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:27.813110 6910 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.814910 6910 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:27.815361 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:27.815975 6301 raft_consensus.cc:3055] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:27.816205 6301 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:27.816840 6301 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.817812 6301 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:27.818256 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:27.818821 6378 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:27.823827 6301 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:27.824769 6378 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:27.824915 6874 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:27.825665 7448 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:27.826068 7448 raft_consensus.cc:697] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:27.826905 7448 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.834568 6130 catalog_manager.cc:5696] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:27.883471 7411 raft_consensus.cc:993] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:27.883916 7411 raft_consensus.cc:1081] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:27.885272 6910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53396
I20260620 19:58:27.885742 6910 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:27.886013 6910 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:27.890324 6910 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.891649 6910 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:27.892364 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:27.892534 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:27.892972 6231 raft_consensus.cc:3055] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:27.892987 6301 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:27.893400 6231 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:27.894202 6231 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.895411 6231 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:27.899207 6301 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:27.900262 6874 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:27.901099 7448 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:27.901595 7448 raft_consensus.cc:697] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:27.901477 6231 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:27.902522 7448 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:27.910526 6130 catalog_manager.cc:5696] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:28.067039 7337 raft_consensus.cc:993] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:28.067548 7504 raft_consensus.cc:1081] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:28.068902 6231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "99e667a051414b87955b78b2520409b5"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:28.069407 6231 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:28.069732 6231 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:28.073809 6231 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:28.075464 6231 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:28.076066 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:28.076275 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:28.076687 6301 raft_consensus.cc:3055] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:28.076858 6378 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:28.077008 6301 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:28.077696 6301 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:28.078589 6301 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:28.081197 6378 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:28.083478 6198 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:28.083608 6301 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:28.085736 7411 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:28.086447 7411 raft_consensus.cc:697] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:28.089572 7411 consensus_queue.cc:237] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:28.097154 6130 catalog_manager.cc:5696] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:28.261911 7506 raft_consensus.cc:993] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:28.262436 7317 raft_consensus.cc:1081] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:28.264120 6299 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:28.264561 6299 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:28.264822 6299 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:28.269577 6299 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:28.271100 6299 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:28.272075 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:28.272425 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:28.272718 6378 raft_consensus.cc:3055] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:28.272964 6226 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:28.273087 6378 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:28.273828 6378 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:28.275007 6378 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:28.277453 6226 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:28.278813 6271 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:28.279603 7504 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:28.280098 6378 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:28.280108 7504 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:28.281154 7504 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:28.291998 6130 catalog_manager.cc:5696] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:28.314340 6226 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:28.315546 7444 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:28.318282 6299 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:28.320247 7448 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:28.327544 6301 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:28.334648 7448 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:28.336645 6378 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:28.339320 7448 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:28.464102 6301 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:28.469679 6378 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:28.469887 6224 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260620 19:58:28.469970 6271 consensus_peers.cc:597] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79a07a8804784405920baab70a535338. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:28.471068 7504 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:28.471900 7472 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:28.479041 7337 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.480573 6224 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.481125 6378 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.494136 6129 catalog_manager.cc:5696] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:28.498910 6178 auto_rebalancer.cc:591] Failed to schedule move for tablet 99e667a051414b87955b78b2520409b5: Illegal state: Leader has not yet committed an operation in its own term
I20260620 19:58:28.501408 6301 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:28.503582 6299 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:28.510118 7411 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:28.511199 6378 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:28.511323 6224 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:28.513139 7337 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260620 19:58:28.515969 6271 consensus_peers.cc:597] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34bcb03937cb4cea98433c0a226855de. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:28.516407 7504 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:28.526659 7543 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.538081 6378 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.546453 6375 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:28.570632 7537 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:28.577723 6128 catalog_manager.cc:5696] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.589915 6231 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.566253 6301 consensus_queue.cc:237] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:28.615068 6271 consensus_peers.cc:597] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eae1222c89dd47e78f503964b6a95973. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:28.623878 6231 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:28.626588 6378 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:28.628722 7504 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:28.634464 7504 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:28.646134 7472 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.648782 6231 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.656975 6377 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.667328 6129 catalog_manager.cc:5696] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:28.685444 6377 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:28.686785 7504 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:28.703764 6224 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:28.707371 7337 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:28.726506 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:28.784235 6377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:28.784737 6377 raft_consensus.cc:606] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:28.787060 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:28.787495 6231 raft_consensus.cc:606] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:28.789870 6299 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:28.790298 6299 raft_consensus.cc:606] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:28.791328 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260620 19:58:28.792510 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:28.793148 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:28.895550 7554 ts_tablet_manager.cc:933] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:28.902678 7554 tablet_copy_client.cc:323] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:28.904122 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 79a07a8804784405920baab70a535338 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:28.904587 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 79a07a8804784405920baab70a535338 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-79a07a8804784405920baab70a535338
I20260620 19:58:28.909322 6317 tablet_copy_source_session.cc:215] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:28.911995 7554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a07a8804784405920baab70a535338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:28.924451 7554 tablet_copy_client.cc:806] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:28.924961 7554 tablet_copy_client.cc:670] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:28.928737 7556 ts_tablet_manager.cc:933] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:28.929435 7554 tablet_copy_client.cc:538] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:28.930706 7556 tablet_copy_client.cc:323] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:28.932214 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 34bcb03937cb4cea98433c0a226855de from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:28.932699 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 34bcb03937cb4cea98433c0a226855de from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-34bcb03937cb4cea98433c0a226855de
I20260620 19:58:28.938143 6317 tablet_copy_source_session.cc:215] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:28.938220 7554 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:28.940572 7556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34bcb03937cb4cea98433c0a226855de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:28.953974 7556 tablet_copy_client.cc:806] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:28.955891 7556 tablet_copy_client.cc:670] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:28.959949 7556 tablet_copy_client.cc:538] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:28.976274 7554 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:28.977311 7554 tablet_bootstrap.cc:492] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:28.978071 7554 ts_tablet_manager.cc:1403] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.008s
I20260620 19:58:28.978880 7529 raft_consensus.cc:993] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:28.979359 7529 raft_consensus.cc:1081] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:28.981240 7554 raft_consensus.cc:359] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:28.982229 7554 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:28.982517 7556 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:28.983079 7554 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:28.986629 7554 ts_tablet_manager.cc:1434] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260620 19:58:28.993059 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-79a07a8804784405920baab70a535338 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:28.993497 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-79a07a8804784405920baab70a535338 on tablet 79a07a8804784405920baab70a535338 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:28.994518 6299 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:28.995153 6299 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:28.995489 6299 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:29.020606 7537 raft_consensus.cc:993] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:29.021147 7537 raft_consensus.cc:1081] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:29.025785 6377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49028
I20260620 19:58:29.026439 6377 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:29.035056 6377 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:29.044163 7554 ts_tablet_manager.cc:933] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:29.052209 7554 tablet_copy_client.cc:323] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:29.058292 6299 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.072852 6299 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:29.075152 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet eae1222c89dd47e78f503964b6a95973 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:29.075706 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet eae1222c89dd47e78f503964b6a95973 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-eae1222c89dd47e78f503964b6a95973
I20260620 19:58:29.077363 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:29.077935 6375 raft_consensus.cc:3055] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:29.078205 6375 raft_consensus.cc:740] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:29.078935 6375 consensus_queue.cc:260] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.080291 6375 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:29.088430 6317 tablet_copy_source_session.cc:215] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:29.095412 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:29.095937 6224 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:29.098227 6375 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:29.099314 6273 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:29.100049 7472 raft_consensus.cc:2804] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:29.100478 7472 raft_consensus.cc:697] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:29.101255 7472 consensus_queue.cc:237] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.101819 6224 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:29.066553 6377 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.105337 7554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae1222c89dd47e78f503964b6a95973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:29.110292 6377 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:29.110438 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:29.115041 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:29.123692 6224 raft_consensus.cc:3055] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:29.123819 6299 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:29.124055 6224 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:29.124827 6224 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.125870 6224 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:29.137326 6224 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:29.138641 6345 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:29.139436 7506 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:29.140903 7506 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:29.146935 7506 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.150767 6299 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:29.177030 7556 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:29.178035 7556 tablet_bootstrap.cc:492] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:29.180343 7556 ts_tablet_manager.cc:1403] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.198s user 0.075s sys 0.077s
I20260620 19:58:29.183310 7556 raft_consensus.cc:359] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:29.184208 7556 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:29.184317 6128 catalog_manager.cc:5696] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:29.185027 7556 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:29.187673 7556 ts_tablet_manager.cc:1434] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:58:29.192382 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-34bcb03937cb4cea98433c0a226855de received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:29.192772 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-34bcb03937cb4cea98433c0a226855de on tablet 34bcb03937cb4cea98433c0a226855de with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:29.209743 7554 tablet_copy_client.cc:806] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:29.210314 7554 tablet_copy_client.cc:670] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:29.214649 7554 tablet_copy_client.cc:538] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:29.225914 7554 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:29.231562 6128 catalog_manager.cc:5696] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:29.249884 7554 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:29.250815 7554 tablet_bootstrap.cc:492] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:29.251447 7554 ts_tablet_manager.cc:1403] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.001s
I20260620 19:58:29.254225 7554 raft_consensus.cc:359] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:29.255030 7554 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:29.255653 7554 consensus_queue.cc:260] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:29.257997 7504 raft_consensus.cc:993] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:29.259397 7472 raft_consensus.cc:1081] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:29.261087 6375 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49014
I20260620 19:58:29.261756 7554 ts_tablet_manager.cc:1434] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260620 19:58:29.261593 6375 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:29.262398 6375 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:29.267131 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-eae1222c89dd47e78f503964b6a95973 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:29.267573 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-eae1222c89dd47e78f503964b6a95973 on tablet eae1222c89dd47e78f503964b6a95973 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:29.273188 6910 raft_consensus.cc:1217] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 19:58:29.274740 6375 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.277844 6375 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:29.280391 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:29.280797 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:29.281353 6224 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:29.280941 6301 raft_consensus.cc:3055] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:29.284066 6301 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:29.284916 6301 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.285966 6301 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:29.287704 6224 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:29.290853 6345 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:29.291710 7506 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:29.292210 7506 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:29.292943 7506 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:29.302884 6301 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:29.304379 6128 catalog_manager.cc:5696] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:29.544396 6378 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:29.546221 7472 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:29.549655 6910 raft_consensus.cc:1217] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 19:58:29.585410 6910 raft_consensus.cc:1217] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:29.586182 6226 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:29.589021 7472 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:29.632109 6226 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:29.633924 7506 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:29.653357 6299 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:29.654767 7506 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:29.809067 6301 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:29.811118 7506 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:29.821501 6224 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:29.829452 7506 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:29.843403 7504 raft_consensus.cc:1064] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:29.850806 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:29.852011 7504 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:29.871176 6907 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:29.888037 7472 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:29.901429 6375 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:29.904107 7472 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:29.906750 6231 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:29.911871 7579 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:29.980907 7579 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:29.990325 6299 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:29.990957 6299 raft_consensus.cc:606] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:29.992357 6910 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:29.993557 6377 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.003496 6375 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:30.004029 6375 raft_consensus.cc:606] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:30.007807 6375 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:30.008334 6375 raft_consensus.cc:606] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:30.007179 6128 catalog_manager.cc:5696] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.010726 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260620 19:58:30.012156 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:30.012864 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:30.019977 6231 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.023279 7472 raft_consensus.cc:1064] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
W20260620 19:58:30.023852 7472 raft_consensus.cc:1068] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Unable to promote non-voter 20bd146dc0cb4917a1a5ce05ec609ce1: Service unavailable: leader transfer in progress
I20260620 19:58:30.030026 7543 raft_consensus.cc:1064] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:30.032405 7543 consensus_queue.cc:237] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:30.074541 7317 raft_consensus.cc:993] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:30.075402 7506 raft_consensus.cc:1081] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:30.080984 6301 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:30.081676 6301 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:30.082056 6301 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:30.084144 6375 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:30.084761 6231 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:30.085549 7337 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:30.086364 7504 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:30.088641 6907 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:30.092711 7337 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:30.095530 6301 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:30.098114 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:30.098845 6226 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:30.099845 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:30.100383 6377 raft_consensus.cc:3055] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:30.100660 6377 raft_consensus.cc:740] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:30.101364 6377 consensus_queue.cc:260] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:30.102306 6377 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:30.110420 6377 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:30.111869 6273 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:30.113227 7337 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:30.113747 7337 raft_consensus.cc:697] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:30.114571 7337 consensus_queue.cc:237] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:30.120898 6299 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:30.098912 6301 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:30.124591 6226 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:30.144658 6227 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:30.144560 6907 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.146131 6909 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:30.148839 7472 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:30.150712 7472 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:30.137784 7579 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.164732 7504 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.166442 6227 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.179219 6231 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.181102 6129 catalog_manager.cc:5696] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:30.181221 6910 raft_consensus.cc:2955] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.304757 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 79a07a8804784405920baab70a535338 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:30.306515 6375 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.328154 7543 raft_consensus.cc:993] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:30.331655 7504 raft_consensus.cc:1081] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:30.336871 6127 catalog_manager.cc:5696] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.337028 6130 catalog_manager.cc:5696] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.344944 7317 raft_consensus.cc:993] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:30.345460 7317 raft_consensus.cc:1081] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:30.347019 6231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45810
I20260620 19:58:30.347530 6231 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:30.347872 6231 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:30.348416 6227 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:30.348915 6227 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:30.349246 6227 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:30.362377 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 79a07a8804784405920baab70a535338 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:30.376143 6227 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:30.378028 6227 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:30.379781 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:30.383445 6307 consensus_queue.cc:237] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:30.384142 6301 raft_consensus.cc:3055] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:30.384442 6301 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:30.385156 6301 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:30.381044 6231 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:30.388967 6910 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:30.390192 7543 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:30.393417 6377 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:30.394165 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:30.394711 6374 raft_consensus.cc:3055] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:30.394907 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:30.395023 6374 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:30.395499 6910 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:30.395852 6374 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:30.396896 6374 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:30.398455 7543 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:30.393707 6231 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:30.404510 6907 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.405769 7597 tablet_replica.cc:333] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:30.406329 6910 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:30.415321 6196 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:30.415019 7597 raft_consensus.cc:2243] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:30.416960 7577 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:30.416697 7597 raft_consensus.cc:2272] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:30.417564 7577 raft_consensus.cc:697] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:30.418479 7577 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:30.379195 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:30.420066 6375 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:30.420331 7597 ts_tablet_manager.cc:1916] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:30.408668 6374 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:30.403873 7337 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.386523 6301 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:30.427492 6377 raft_consensus.cc:2955] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:30.445149 7597 ts_tablet_manager.cc:1929] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260620 19:58:30.445658 7597 log.cc:1199] T 79a07a8804784405920baab70a535338 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/79a07a8804784405920baab70a535338
I20260620 19:58:30.447214 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 79a07a8804784405920baab70a535338 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:30.446933 6127 catalog_manager.cc:5696] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:30.452128 6301 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:30.453871 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet eae1222c89dd47e78f503964b6a95973 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:30.460011 6195 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:30.460837 7577 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:30.461280 7577 raft_consensus.cc:697] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:30.461256 6129 catalog_manager.cc:5696] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:30.462368 7577 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:30.467129 6375 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:30.478044 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet eae1222c89dd47e78f503964b6a95973 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:30.490546 7600 tablet_replica.cc:333] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:30.490196 6127 catalog_manager.cc:5696] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:30.493022 7600 raft_consensus.cc:2243] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:30.493736 7600 raft_consensus.cc:2272] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:30.497008 7600 ts_tablet_manager.cc:1916] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:30.512301 7600 ts_tablet_manager.cc:1929] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:30.512748 7600 log.cc:1199] T eae1222c89dd47e78f503964b6a95973 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/eae1222c89dd47e78f503964b6a95973
I20260620 19:58:30.514453 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet eae1222c89dd47e78f503964b6a95973 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:30.523128 6377 raft_consensus.cc:1275] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:30.524189 7337 consensus_queue.cc:1048] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:30.559770 6227 raft_consensus.cc:1275] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:30.567648 7579 consensus_queue.cc:1048] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:30.898695 6377 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:30.903312 6377 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:30.904465 7537 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:30.904732 7561 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:30.927350 6301 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:30.929972 7537 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:30.933508 6907 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:30.935911 7537 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:30.939540 6910 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Advancing to term 3
I20260620 19:58:30.956396 6910 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:30.957747 7537 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:30.976208 7561 raft_consensus.cc:1064] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:30.978539 7561 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:30.986320 6908 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:30.989348 6377 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:30.990111 7561 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:30.992552 7577 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:31.007176 7577 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.011734 6306 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:31.021220 7537 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:31.025228 6306 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.032641 6377 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.043459 6127 catalog_manager.cc:5696] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 3 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:31.043701 6910 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.315330 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:31.375202 6227 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:31.375779 6227 raft_consensus.cc:606] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:31.378521 6306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a56f959cee394e398810b9170b9d24be"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:31.382721 6306 raft_consensus.cc:606] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:31.387475 6227 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:31.388005 6227 raft_consensus.cc:606] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:31.389487 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260620 19:58:31.391420 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:31.392122 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:31.439540 6227 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:31.444684 6910 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:31.446228 7561 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:31.447391 6306 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:31.449445 7577 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:31.454078 6910 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.452519 7577 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.476408 6306 raft_consensus.cc:2955] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.489108 6127 catalog_manager.cc:5696] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 5 to 6, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:31.491043 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 34bcb03937cb4cea98433c0a226855de with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:31.506464 7561 raft_consensus.cc:993] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:31.507061 7561 raft_consensus.cc:1081] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:31.507926 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 34bcb03937cb4cea98433c0a226855de with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:31.509380 6307 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52350
I20260620 19:58:31.509884 6307 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:31.510216 6307 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:31.513159 7619 tablet_replica.cc:333] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:31.514133 7619 raft_consensus.cc:2243] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:31.514910 7619 raft_consensus.cc:2272] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:31.516474 6307 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.518241 6307 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:31.518633 7619 ts_tablet_manager.cc:1916] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:31.519285 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:31.519727 6377 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:31.518811 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:31.523816 6227 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:31.524127 6227 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:31.524852 6227 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.526003 6227 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:31.526319 6377 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:31.527411 6273 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:31.527997 7617 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:31.528474 7617 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:31.529242 7617 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.535547 7619 ts_tablet_manager.cc:1929] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260620 19:58:31.536000 7619 log.cc:1199] T 34bcb03937cb4cea98433c0a226855de P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/34bcb03937cb4cea98433c0a226855de
I20260620 19:58:31.537406 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 34bcb03937cb4cea98433c0a226855de (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:31.540313 6128 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:31.541464 6227 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:31.634362 7579 raft_consensus.cc:993] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:31.634824 7617 raft_consensus.cc:1081] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:31.636233 6375 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a56f959cee394e398810b9170b9d24be"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49014
I20260620 19:58:31.636633 6375 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:31.636855 6375 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:31.641613 6375 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.643575 6375 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:31.644613 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:31.647796 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:31.647956 6307 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:58:31.648648 6307 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:31.649418 6307 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.650588 6307 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:31.648344 6227 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:31.656996 6307 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:58:31.657913 6227 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:58:31.658252 6344 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:31.659456 7618 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:58:31.659929 7618 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:31.661275 7618 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.674063 6128 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 5 to 6, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 6 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:31.837734 7537 raft_consensus.cc:993] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:31.839210 7537 raft_consensus.cc:1081] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:31.840720 6377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49028
I20260620 19:58:31.841198 6377 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:31.841527 6377 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:31.846750 6377 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.848052 6377 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:31.848943 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:31.849493 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:31.849486 6306 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:31.850013 6227 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:31.850291 6227 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:31.850960 6227 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.851794 6227 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:31.854959 6306 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:31.855955 6344 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:31.856437 6227 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:31.856618 7608 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:31.857177 7608 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:31.857961 7608 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:31.867400 6128 catalog_manager.cc:5696] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 3, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:31.974874 6377 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:31.980628 7579 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:32.013830 6231 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:32.015602 7628 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:32.078265 6306 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260620 19:58:32.080144 7618 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:32.091667 6231 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260620 19:58:32.093258 7618 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:32.257335 6301 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:32.258734 7608 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:32.270867 6231 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:32.273082 7618 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:32.472550 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:32.513789 6377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:32.514559 6377 raft_consensus.cc:606] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:32.522553 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:32.523126 6307 raft_consensus.cc:606] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:32.525612 6377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:32.526089 6377 raft_consensus.cc:606] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:32.527052 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260620 19:58:32.528618 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:32.529601 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:32.590945 6377 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:32.597121 6307 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:32.597227 6231 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:32.597607 6345 consensus_peers.cc:597] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aacb05a304e442a7bd73148aff5f4a66. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:32.598453 7633 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:32.600137 7618 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:32.609436 7608 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.614037 6307 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.614977 6231 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.630820 6129 catalog_manager.cc:5696] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.649447 6375 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:32.656951 6231 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260620 19:58:32.657168 6345 consensus_peers.cc:597] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a56f959cee394e398810b9170b9d24be. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:32.658046 6307 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260620 19:58:32.658363 7608 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:32.660374 7641 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:32.667874 7628 raft_consensus.cc:993] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:32.667133 7633 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.668457 7617 raft_consensus.cc:1081] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:32.671084 6227 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:32.671659 6227 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:32.672008 6227 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:32.672158 6231 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.676266 6301 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index -1 to 6, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.679278 6227 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.686619 6227 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:32.687009 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:32.687110 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:32.687754 6307 raft_consensus.cc:3055] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:32.687906 6375 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:32.688093 6307 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:32.688907 6307 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.689936 6307 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:32.693099 6375 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:32.692975 6129 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 6, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 6 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.694259 6198 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:32.695080 7613 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:32.695518 7613 raft_consensus.cc:697] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:32.696273 6307 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:32.696456 7613 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.711047 6227 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:32.725306 6196 consensus_peers.cc:597] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b0538f91fad4439a7d30adbf2f83b57. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:32.732208 6301 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:32.730481 6129 catalog_manager.cc:5696] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:32.734786 7613 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:32.735845 6375 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:32.737440 7613 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:32.739050 7645 raft_consensus.cc:993] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:32.739498 7645 raft_consensus.cc:1081] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:32.741400 6231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45810
I20260620 19:58:32.741978 6231 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:32.742377 6231 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:32.759191 6231 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.759011 7537 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.766551 6231 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:32.767144 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:32.767697 6377 raft_consensus.cc:3055] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:32.767948 6377 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:32.768621 6377 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.769805 6377 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:32.759321 6375 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.775413 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:32.775979 6307 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:32.780323 6377 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:32.783604 6198 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:32.785018 7613 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:32.785584 7613 raft_consensus.cc:697] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:32.786872 6307 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:32.787068 7613 consensus_queue.cc:237] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.813496 6307 consensus_queue.cc:237] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:32.814179 6127 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.820823 6129 catalog_manager.cc:5696] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:32.823016 6301 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.826673 7633 raft_consensus.cc:993] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:32.827178 7633 raft_consensus.cc:1081] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:32.829421 6306 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:32.829947 6306 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:32.830371 6306 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:32.841223 6306 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.849189 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:32.849905 6231 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:32.855407 6377 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:32.858167 6306 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:32.860193 6226 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:32.860707 6231 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:32.860790 7617 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:32.862839 7624 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:32.863041 6271 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:32.863835 7617 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
W20260620 19:58:32.863960 6271 consensus_peers.cc:597] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31d307880255440b9b2c96eee7bb9f28. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:32.864557 7617 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:32.865656 7617 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.859503 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:32.866734 6375 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:32.867090 6375 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:32.868033 6375 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:32.871063 6375 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:32.871148 7617 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.878739 6377 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.882179 6375 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:32.881472 6127 catalog_manager.cc:5696] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 3 to 4, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 4 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:32.903539 6231 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:32.908365 6128 catalog_manager.cc:5696] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:33.051831 7657 ts_tablet_manager.cc:933] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:33.054812 7657 tablet_copy_client.cc:323] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:33.056493 6391 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet a56f959cee394e398810b9170b9d24be from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:33.056981 6391 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet a56f959cee394e398810b9170b9d24be from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-a56f959cee394e398810b9170b9d24be
I20260620 19:58:33.063941 6391 tablet_copy_source_session.cc:215] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:33.066531 7657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56f959cee394e398810b9170b9d24be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:33.078860 7657 tablet_copy_client.cc:806] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:33.080199 7657 tablet_copy_client.cc:670] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:33.083832 7657 tablet_copy_client.cc:538] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:33.090363 7657 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:33.113401 7657 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:33.114169 7657 tablet_bootstrap.cc:492] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:33.114820 7657 ts_tablet_manager.cc:1403] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.025s user 0.026s sys 0.001s
I20260620 19:58:33.118621 7657 raft_consensus.cc:359] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.120005 7657 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:33.120896 7657 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.121511 7660 ts_tablet_manager.cc:933] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:33.123390 7657 ts_tablet_manager.cc:1434] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260620 19:58:33.124473 7660 tablet_copy_client.cc:323] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:33.125468 6391 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a56f959cee394e398810b9170b9d24be received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:33.125973 6391 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-a56f959cee394e398810b9170b9d24be on tablet a56f959cee394e398810b9170b9d24be with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:33.127027 6390 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet aacb05a304e442a7bd73148aff5f4a66 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:33.127560 6390 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet aacb05a304e442a7bd73148aff5f4a66 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-aacb05a304e442a7bd73148aff5f4a66
I20260620 19:58:33.135474 6390 tablet_copy_source_session.cc:215] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:33.139901 7660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacb05a304e442a7bd73148aff5f4a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:33.156208 7660 tablet_copy_client.cc:806] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:33.156868 7660 tablet_copy_client.cc:670] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:33.161641 7660 tablet_copy_client.cc:538] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:33.170120 7660 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:33.189985 7657 ts_tablet_manager.cc:933] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:33.192188 7660 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:33.193089 7660 tablet_bootstrap.cc:492] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:33.193745 7660 ts_tablet_manager.cc:1403] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.024s user 0.026s sys 0.000s
I20260620 19:58:33.196810 7660 raft_consensus.cc:359] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.197642 7660 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:33.198277 7660 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.200762 7660 ts_tablet_manager.cc:1434] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:58:33.203274 6390 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-aacb05a304e442a7bd73148aff5f4a66 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:33.203621 6390 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-aacb05a304e442a7bd73148aff5f4a66 on tablet aacb05a304e442a7bd73148aff5f4a66 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:33.205948 6307 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:33.210196 7657 tablet_copy_client.cc:323] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:33.211089 7650 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:33.212215 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 8b0538f91fad4439a7d30adbf2f83b57 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:33.212726 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 8b0538f91fad4439a7d30adbf2f83b57 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-8b0538f91fad4439a7d30adbf2f83b57
I20260620 19:58:33.221999 6242 tablet_copy_source_session.cc:215] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:33.241835 6375 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:33.242064 6377 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:33.248883 7650 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:33.253535 7652 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:33.253980 6306 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:33.261549 7650 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:33.265791 7657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0538f91fad4439a7d30adbf2f83b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:33.294749 7657 tablet_copy_client.cc:806] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:33.295413 7657 tablet_copy_client.cc:670] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:33.299583 7657 tablet_copy_client.cc:538] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:33.308524 7657 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:33.341892 7660 ts_tablet_manager.cc:933] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:33.351933 7657 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:33.353289 7657 tablet_bootstrap.cc:492] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:33.354043 7657 ts_tablet_manager.cc:1403] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.012s
I20260620 19:58:33.357270 7657 raft_consensus.cc:359] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.358158 7657 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:33.373440 7660 tablet_copy_client.cc:323] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:33.376510 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 31d307880255440b9b2c96eee7bb9f28 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:33.377038 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 31d307880255440b9b2c96eee7bb9f28 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-31d307880255440b9b2c96eee7bb9f28
I20260620 19:58:33.358839 7657 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.388379 7657 ts_tablet_manager.cc:1434] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.034s user 0.002s sys 0.003s
I20260620 19:58:33.389962 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8b0538f91fad4439a7d30adbf2f83b57 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:33.390405 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-8b0538f91fad4439a7d30adbf2f83b57 on tablet 8b0538f91fad4439a7d30adbf2f83b57 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:33.393003 6317 tablet_copy_source_session.cc:215] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:33.398964 7660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d307880255440b9b2c96eee7bb9f28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:33.404026 6226 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:33.405386 7624 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:33.415352 7660 tablet_copy_client.cc:806] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:33.416052 7660 tablet_copy_client.cc:670] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:33.426573 6375 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:33.442399 7660 tablet_copy_client.cc:538] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:33.443027 7624 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:33.454053 7660 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:33.495828 7660 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:33.496726 7660 tablet_bootstrap.cc:492] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:33.499614 7660 ts_tablet_manager.cc:1403] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.046s user 0.022s sys 0.008s
I20260620 19:58:33.503451 7660 raft_consensus.cc:359] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.504283 7660 raft_consensus.cc:740] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:33.504863 7660 consensus_queue.cc:260] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:33.507010 7660 ts_tablet_manager.cc:1434] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 19:58:33.508801 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-31d307880255440b9b2c96eee7bb9f28 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:33.509224 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-31d307880255440b9b2c96eee7bb9f28 on tablet 31d307880255440b9b2c96eee7bb9f28 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:33.530923 6909 raft_consensus.cc:1217] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEARNER]: Deduplicated request from leader. Original: 6.5->[6.6-6.6] Dedup: 6.6->[]
I20260620 19:58:33.570932 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:33.619108 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:33.619622 6231 raft_consensus.cc:606] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:33.622095 6306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "64d372664b48432eafee6464408fcef5"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:33.622642 6306 raft_consensus.cc:606] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:33.625015 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:33.625437 6231 raft_consensus.cc:606] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:33.626433 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260620 19:58:33.627801 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:33.628520 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:33.711422 6909 raft_consensus.cc:1217] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:33.720810 6909 raft_consensus.cc:1217] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 19:58:33.860111 6910 raft_consensus.cc:1217] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 19:58:34.000168 7537 raft_consensus.cc:993] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:34.000662 7650 raft_consensus.cc:1081] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:34.001935 6306 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52350
I20260620 19:58:34.002375 6306 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:34.002691 6306 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:34.010488 6306 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:34.012193 6306 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:34.013012 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:34.013409 6375 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:34.013823 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:34.014346 6231 raft_consensus.cc:3055] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:34.014647 6231 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:34.015368 6231 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:34.016618 6231 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:34.022927 6375 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:34.023553 7676 raft_consensus.cc:993] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:34.024010 7676 raft_consensus.cc:1081] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:34.023226 6231 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:34.025288 6273 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:34.026340 7617 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:34.026845 7617 raft_consensus.cc:697] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:34.027392 6375 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "64d372664b48432eafee6464408fcef5"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49014
I20260620 19:58:34.027999 6375 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:34.027716 7617 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:34.028399 6375 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:34.040102 6129 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:34.040890 6375 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:34.043009 6375 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:34.043665 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:34.044373 6306 raft_consensus.cc:3055] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:34.044657 6306 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:34.045382 6306 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:34.046380 6306 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:34.047415 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:34.047960 6231 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:34.054260 6231 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:34.055505 6345 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:34.056303 7633 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:34.056780 7633 raft_consensus.cc:697] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:34.057614 7633 consensus_queue.cc:237] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:34.061038 6306 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:34.073583 7650 raft_consensus.cc:993] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:34.074044 7650 raft_consensus.cc:1081] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:34.075532 6377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49028
I20260620 19:58:34.075371 6127 catalog_manager.cc:5696] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 4 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:34.076116 6377 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:34.076581 6377 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.080861 6306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:34.081393 6306 raft_consensus.cc:606] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Received request to transfer leadership
I20260620 19:58:34.084189 7667 raft_consensus.cc:1064] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:34.086290 7667 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.087423 6306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:34.087924 6306 raft_consensus.cc:606] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Received request to transfer leadership
W20260620 19:58:34.089272 6114 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 8b0538f91fad4439a7d30adbf2f83b57 already in progress [suppressed 33 similar messages]
I20260620 19:58:34.091279 6377 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:34.093477 6377 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:34.095099 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:34.095602 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:34.095896 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:34.096140 6307 raft_consensus.cc:606] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Received request to transfer leadership
I20260620 19:58:34.096411 6306 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.095602 6231 raft_consensus.cc:3055] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:34.097124 6231 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:34.097878 6231 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:34.099030 6231 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
W20260620 19:58:34.100806 6178 auto_rebalancer.cc:855] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260620 19:58:34.101266 6178 auto_rebalancer.cc:863] Could not move replica: Remote error: Service unavailable: leadership transfer for 8b0538f91fad4439a7d30adbf2f83b57 already in progress
W20260620 19:58:34.101539 6178 auto_rebalancer.cc:301] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 8b0538f91fad4439a7d30adbf2f83b57 already in progress
I20260620 19:58:34.105299 6231 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:34.106320 6345 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:34.107120 7686 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:34.107702 7686 raft_consensus.cc:697] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:34.107960 6306 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:34.108978 7686 consensus_queue.cc:237] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:34.112546 7633 raft_consensus.cc:1064] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:34.115200 7633 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.127104 6127 catalog_manager.cc:5696] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:34.136562 6306 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:58:34.138628 7645 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:58:34.142338 6910 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:58:34.150301 6231 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:58:34.165198 6307 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:34.165720 7690 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:58:34.167853 7667 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:34.168721 7633 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:34.174142 6231 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:34.178331 6910 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:34.180173 7688 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:34.182974 7690 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:34.207262 7667 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.216187 6227 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.239142 6307 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.243252 7690 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.244922 6226 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.254889 6127 catalog_manager.cc:5696] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:34.255292 6307 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.262581 6907 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.262878 6909 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 6 to 7, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.282142 6128 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 6 to 7, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.302796 7624 raft_consensus.cc:1064] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:34.305289 7624 consensus_queue.cc:237] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.308259 6377 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.309046 6375 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.315770 6226 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:34.317318 7633 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:34.320556 6910 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:34.323369 7617 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:34.323653 6375 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:34.324177 6227 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:58:34.325163 7676 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:34.325521 6910 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:58:34.326041 6227 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:34.327316 7676 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:34.327602 7693 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:58:34.329207 6907 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:34.330365 7690 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:58:34.345436 7667 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.350826 7688 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.360752 6231 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.362478 7693 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:34.359508 7624 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.364971 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet aacb05a304e442a7bd73148aff5f4a66 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:34.367182 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet a56f959cee394e398810b9170b9d24be with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:34.368193 6226 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.368830 6224 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.371747 6909 raft_consensus.cc:2955] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.377882 6907 raft_consensus.cc:2955] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Committing config change with OpId 6.8: config changed from index 7 to 8, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.399590 6126 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 7 to 8, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 6 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.387570 6129 catalog_manager.cc:5696] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.415320 6130 catalog_manager.cc:5696] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:34.419741 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet a56f959cee394e398810b9170b9d24be with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:34.442803 6377 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.445246 7705 tablet_replica.cc:333] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:34.453498 7705 raft_consensus.cc:2243] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:34.454226 7705 raft_consensus.cc:2272] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:34.458206 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet aacb05a304e442a7bd73148aff5f4a66 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:34.465870 6909 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.467202 7705 ts_tablet_manager.cc:1916] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:34.488524 7705 ts_tablet_manager.cc:1929] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.7
I20260620 19:58:34.488951 7705 log.cc:1199] T a56f959cee394e398810b9170b9d24be P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/a56f959cee394e398810b9170b9d24be
I20260620 19:58:34.494140 7705 tablet_replica.cc:333] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:34.495165 7705 raft_consensus.cc:2243] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:34.495894 7705 raft_consensus.cc:2272] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:34.493055 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet a56f959cee394e398810b9170b9d24be (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:34.509433 7705 ts_tablet_manager.cc:1916] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:34.514025 6307 consensus_queue.cc:237] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.522840 6910 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:34.523280 6226 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:34.524619 7624 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:34.530911 7617 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:34.534677 7705 ts_tablet_manager.cc:1929] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:34.541949 7705 log.cc:1199] T aacb05a304e442a7bd73148aff5f4a66 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/aacb05a304e442a7bd73148aff5f4a66
I20260620 19:58:34.543111 6231 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:34.543682 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet aacb05a304e442a7bd73148aff5f4a66 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:34.542970 7676 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.548868 6910 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.551827 7617 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:34.560694 6226 raft_consensus.cc:2955] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.575035 6909 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Advancing to term 3
I20260620 19:58:34.574553 6129 catalog_manager.cc:5696] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:34.585978 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 31d307880255440b9b2c96eee7bb9f28 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:34.589483 6374 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:34.591025 6909 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:34.592691 7706 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:34.596056 7624 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:34.603717 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 31d307880255440b9b2c96eee7bb9f28 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:34.610157 7706 raft_consensus.cc:993] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:34.610733 7617 raft_consensus.cc:1081] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:34.613215 6231 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:34.615038 7667 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:34.628798 7708 tablet_replica.cc:333] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:34.632174 7708 raft_consensus.cc:2243] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:34.634433 6224 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:34.636494 7686 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:34.636638 7708 raft_consensus.cc:2272] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:34.646710 7708 ts_tablet_manager.cc:1916] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:34.665050 6226 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:34.665998 6226 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:34.666353 6226 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:34.667935 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:34.679000 6307 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:34.690114 6307 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:34.692248 7688 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:34.695161 7688 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:34.704095 6226 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:34.721313 7712 raft_consensus.cc:1064] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
W20260620 19:58:34.721729 7712 raft_consensus.cc:1068] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Unable to promote non-voter 20bd146dc0cb4917a1a5ce05ec609ce1: Service unavailable: leader transfer in progress
I20260620 19:58:34.738010 7624 raft_consensus.cc:1064] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
W20260620 19:58:34.738409 7624 raft_consensus.cc:1068] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Unable to promote non-voter 20bd146dc0cb4917a1a5ce05ec609ce1: Service unavailable: leader transfer in progress
I20260620 19:58:34.739883 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:34.740427 6301 raft_consensus.cc:3055] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:34.740680 6301 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:34.741320 6301 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:34.742583 6301 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:34.749749 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:34.750319 6374 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:34.753203 6301 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:34.754302 6195 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:34.755062 7652 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:34.755519 7652 raft_consensus.cc:697] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:34.756346 7652 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:34.757799 6374 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:34.760406 6226 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:34.767405 7708 ts_tablet_manager.cc:1929] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260620 19:58:34.767794 7708 log.cc:1199] T 31d307880255440b9b2c96eee7bb9f28 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/31d307880255440b9b2c96eee7bb9f28
I20260620 19:58:34.769377 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 31d307880255440b9b2c96eee7bb9f28 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:34.772707 6129 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:34.790436 6374 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03268245eb53439a9b88defe52240125"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:34.791014 6374 raft_consensus.cc:606] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:34.793471 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92212144534e4394844003016ba67a47"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:34.794010 6307 raft_consensus.cc:606] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:34.800275 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:34.800769 6307 raft_consensus.cc:606] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:34.803036 6374 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:34.803546 6374 raft_consensus.cc:606] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:34.805143 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260620 19:58:34.806325 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:34.807008 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:34.861407 7624 raft_consensus.cc:993] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:34.861860 7712 raft_consensus.cc:1081] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:34.863101 6226 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c686486d28e847d0a66123f171f1149f"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:34.863473 6226 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:34.863687 6226 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:34.869619 6226 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:34.871212 6226 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:34.872030 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:34.872030 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:34.872793 6375 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:34.872891 6307 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:34.873247 6307 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:34.873833 6307 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:34.874671 6307 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:34.878023 6375 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:34.878927 6198 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:34.879508 7650 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:34.879964 7650 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:34.881042 6307 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:34.880721 7650 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:34.889963 6128 catalog_manager.cc:5696] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 4 to 5, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:34.949719 7686 raft_consensus.cc:993] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:34.950192 7645 raft_consensus.cc:1081] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:34.951949 6910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53406
I20260620 19:58:34.952396 6910 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:34.952739 6910 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.956413 7686 raft_consensus.cc:993] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:34.956899 7645 raft_consensus.cc:1081] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:34.958412 6301 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "03268245eb53439a9b88defe52240125"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:34.958914 6301 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:34.959210 6301 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.960258 6910 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.962215 6910 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:34.963269 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:34.963800 6375 raft_consensus.cc:3055] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:34.964071 6375 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:34.964751 6375 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.965790 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:34.966380 6307 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.966075 6301 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.966483 6375 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.969190 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:34.969696 6374 raft_consensus.cc:3055] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:34.969982 6374 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:34.970729 6374 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.971732 6374 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.972780 6375 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:34.972780 6307 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:34.974146 6877 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:34.974892 7631 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:34.974416 6301 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:34.975378 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:34.975951 6910 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:34.975407 7631 raft_consensus.cc:697] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:34.977321 6374 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:34.977221 7631 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.978336 6273 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:34.979055 7624 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:34.979518 7624 raft_consensus.cc:697] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:34.980374 7624 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:34.982127 6910 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:34.988039 6128 catalog_manager.cc:5696] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:34.993758 6129 catalog_manager.cc:5696] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:34.999096 7624 raft_consensus.cc:993] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:34.999548 7712 raft_consensus.cc:1081] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:35.002350 6226 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92212144534e4394844003016ba67a47"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:35.002802 6226 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:35.003073 6226 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:35.007265 6226 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:35.008657 6226 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:35.009275 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:35.009485 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:35.009835 6301 raft_consensus.cc:3055] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:35.009943 6374 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:35.010176 6301 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:35.010959 6301 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:35.012086 6301 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:35.016070 6374 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:35.017302 6198 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:35.018164 7650 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:35.018287 6301 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:35.020143 7650 raft_consensus.cc:697] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:35.021085 7650 consensus_queue.cc:237] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:35.037391 6129 catalog_manager.cc:5696] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:35.161357 6375 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260620 19:58:35.162339 7652 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.169271 6910 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Advancing to term 4
I20260620 19:58:35.176590 6910 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260620 19:58:35.178642 7650 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.181784 6307 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260620 19:58:35.183646 7650 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.193635 7728 raft_consensus.cc:1064] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:35.196033 7728 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:35.203397 6375 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:35.203572 6910 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:35.205524 7650 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:35.206200 7726 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:35.207376 6301 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:35.209069 7728 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:35.217597 7726 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 3 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.219692 6907 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.227222 6301 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.230360 6374 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.239305 6129 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.259611 6231 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:35.266016 6910 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 19:58:35.266567 6375 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 19:58:35.267400 7650 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:58:35.268486 7728 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:35.274267 7728 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.277788 6375 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.277814 6907 raft_consensus.cc:2955] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:35.287344 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 8b0538f91fad4439a7d30adbf2f83b57 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:35.292311 6128 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:35.303960 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 8b0538f91fad4439a7d30adbf2f83b57 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:35.310034 7729 tablet_replica.cc:333] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:35.310834 7729 raft_consensus.cc:2243] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:35.311446 7729 raft_consensus.cc:2272] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:35.315673 7729 ts_tablet_manager.cc:1916] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:35.330029 7729 ts_tablet_manager.cc:1929] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20260620 19:58:35.330397 7729 log.cc:1199] T 8b0538f91fad4439a7d30adbf2f83b57 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/8b0538f91fad4439a7d30adbf2f83b57
I20260620 19:58:35.331876 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 8b0538f91fad4439a7d30adbf2f83b57 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:35.362015 6375 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260620 19:58:35.363272 7728 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.374782 6307 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260620 19:58:35.376096 7650 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.443028 6375 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:35.444173 7704 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.452243 6307 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:35.453593 7733 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.462491 6907 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:35.463907 7711 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.464704 6301 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:35.467121 7726 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:35.478870 6375 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:35.480603 7706 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:35.496333 6374 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:35.499696 7726 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:35.652470 6375 consensus_queue.cc:237] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:35.658871 6301 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:35.660465 7686 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:35.661623 6231 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:35.663750 6345 consensus_peers.cc:597] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af9232a7f82e4c3f911b856592bd6ba9. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:35.664955 7645 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:35.668357 7645 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.670240 6301 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.683461 6226 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.687436 6231 consensus_queue.cc:237] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:35.690732 6130 catalog_manager.cc:5696] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:35.706990 6196 consensus_peers.cc:597] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e617e76f53974a7189d97fe9f8e69a8b. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:35.707738 6307 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:35.710623 7652 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:35.712383 6375 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:35.715173 7726 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260620 19:58:35.720091 7652 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.721510 6306 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.734985 6130 catalog_manager.cc:5696] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.737042 6226 consensus_queue.cc:237] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:35.742501 6375 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.745950 6374 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:35.746915 6196 consensus_peers.cc:597] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc6cbf763dbc4c3888feddf378ef4405. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:35.747840 7728 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:35.750458 6306 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:35.751892 7728 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:35.760402 7728 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.767431 6306 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.771955 6374 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.785902 6130 catalog_manager.cc:5696] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.791904 6306 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:35.803242 6271 consensus_peers.cc:597] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee78433ca73b4ea8b2b24535036f6369. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:35.807130 6226 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:35.807130 6374 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:35.808578 7706 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:35.809361 7712 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:35.817307 7712 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.819092 6226 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.833340 6130 catalog_manager.cc:5696] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:35.820024 6374 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:35.846828 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:35.921222 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:35.921840 6231 raft_consensus.cc:606] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:35.924535 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1cb7a18e80e145f6913b635487750934"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:35.925037 6231 raft_consensus.cc:606] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:35.928822 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "99e667a051414b87955b78b2520409b5"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:35.929328 6231 raft_consensus.cc:606] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:35.930970 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260620 19:58:35.932556 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:35.933310 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:36.089573 7748 raft_consensus.cc:993] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:36.090095 7748 raft_consensus.cc:1081] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:36.095546 6301 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "99e667a051414b87955b78b2520409b5"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52350
I20260620 19:58:36.096237 6301 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:36.096571 6301 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:36.103012 6301 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:36.106483 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:36.106851 6301 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:36.107158 6231 raft_consensus.cc:3055] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:36.107645 6231 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:36.108605 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:36.108498 6231 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:36.109112 6375 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:36.109787 6231 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:36.118667 6231 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:36.119890 6271 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:36.120723 7706 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:36.121443 7706 raft_consensus.cc:697] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:36.122394 7706 consensus_queue.cc:237] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:36.120513 6375 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:36.141829 6130 catalog_manager.cc:5696] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 3 to 4, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 4 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:36.154788 7753 ts_tablet_manager.cc:933] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:36.158527 7753 tablet_copy_client.cc:323] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:36.158686 7728 raft_consensus.cc:993] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:36.159228 7728 raft_consensus.cc:1081] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:36.160727 6910 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53396
I20260620 19:58:36.161001 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet e617e76f53974a7189d97fe9f8e69a8b from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:36.161530 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet e617e76f53974a7189d97fe9f8e69a8b from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-e617e76f53974a7189d97fe9f8e69a8b
I20260620 19:58:36.161269 6910 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:36.162482 6910 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:36.162698 7650 raft_consensus.cc:993] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:36.163156 7650 raft_consensus.cc:1081] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:36.164650 6375 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1cb7a18e80e145f6913b635487750934"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49028
I20260620 19:58:36.165133 6375 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:36.165450 6375 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:36.168800 6242 tablet_copy_source_session.cc:215] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:36.169214 6910 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:36.172026 6910 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:36.173446 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:36.173542 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:36.174013 6306 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:36.174119 6231 raft_consensus.cc:3055] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:36.174546 6231 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:36.174465 6375 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:36.175763 7753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e617e76f53974a7189d97fe9f8e69a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:36.175463 6231 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:36.177071 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:36.177533 6231 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:36.177589 6224 raft_consensus.cc:3055] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:36.177812 6375 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:36.178028 6224 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:36.178774 6224 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:36.177702 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:36.179915 6910 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:36.183557 6306 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:36.184796 6874 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:36.185477 7704 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:36.185930 7704 raft_consensus.cc:697] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:36.186045 6910 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:36.186872 7704 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:36.179924 6224 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:36.187263 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:36.189018 7645 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:36.189473 7645 raft_consensus.cc:697] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:36.190249 7645 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:36.194240 6224 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:36.194442 7753 tablet_copy_client.cc:806] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:36.207048 7753 tablet_copy_client.cc:670] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:36.200043 6231 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:36.201251 7755 ts_tablet_manager.cc:933] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:36.213260 6130 catalog_manager.cc:5696] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:36.218706 7753 tablet_copy_client.cc:538] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:36.224615 6128 catalog_manager.cc:5696] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:36.227563 7755 tablet_copy_client.cc:323] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:36.229128 6390 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet af9232a7f82e4c3f911b856592bd6ba9 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:58:36.229650 6390 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet af9232a7f82e4c3f911b856592bd6ba9 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-af9232a7f82e4c3f911b856592bd6ba9
I20260620 19:58:36.231094 7753 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:36.247723 6390 tablet_copy_source_session.cc:215] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:36.251861 7755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9232a7f82e4c3f911b856592bd6ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:36.260557 7753 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:36.261451 7753 tablet_bootstrap.cc:492] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:36.262090 7753 ts_tablet_manager.cc:1403] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.005s
I20260620 19:58:36.265151 7753 raft_consensus.cc:359] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.265939 7753 raft_consensus.cc:740] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:36.266570 7753 consensus_queue.cc:260] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.267132 7755 tablet_copy_client.cc:806] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:36.267746 7755 tablet_copy_client.cc:670] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:36.270504 7753 ts_tablet_manager.cc:1434] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.008s user 0.003s sys 0.008s
I20260620 19:58:36.275915 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-e617e76f53974a7189d97fe9f8e69a8b received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:36.276333 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-e617e76f53974a7189d97fe9f8e69a8b on tablet e617e76f53974a7189d97fe9f8e69a8b with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:36.278589 7755 tablet_copy_client.cc:538] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:36.283946 7753 ts_tablet_manager.cc:933] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:36.287192 7755 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:36.299978 7753 tablet_copy_client.cc:323] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:36.301730 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet ee78433ca73b4ea8b2b24535036f6369 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:36.302184 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet ee78433ca73b4ea8b2b24535036f6369 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-ee78433ca73b4ea8b2b24535036f6369
I20260620 19:58:36.312336 7755 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:36.312822 6317 tablet_copy_source_session.cc:215] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:36.313251 7755 tablet_bootstrap.cc:492] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:36.314972 7762 ts_tablet_manager.cc:933] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:36.315052 7755 ts_tablet_manager.cc:1403] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.000s
I20260620 19:58:36.316051 7753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee78433ca73b4ea8b2b24535036f6369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:36.319873 7755 raft_consensus.cc:359] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.320967 7755 raft_consensus.cc:740] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:36.321803 7755 consensus_queue.cc:260] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.324807 7755 ts_tablet_manager.cc:1434] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.009s user 0.002s sys 0.004s
I20260620 19:58:36.326876 7762 tablet_copy_client.cc:323] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:36.329073 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet dc6cbf763dbc4c3888feddf378ef4405 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:36.329566 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet dc6cbf763dbc4c3888feddf378ef4405 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-dc6cbf763dbc4c3888feddf378ef4405
I20260620 19:58:36.333372 6390 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-af9232a7f82e4c3f911b856592bd6ba9 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:58:36.333774 6390 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-af9232a7f82e4c3f911b856592bd6ba9 on tablet af9232a7f82e4c3f911b856592bd6ba9 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:36.338657 6242 tablet_copy_source_session.cc:215] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:36.347875 7762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6cbf763dbc4c3888feddf378ef4405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:36.354306 7753 tablet_copy_client.cc:806] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:36.354919 7753 tablet_copy_client.cc:670] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:36.364488 7762 tablet_copy_client.cc:806] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:36.365964 7762 tablet_copy_client.cc:670] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:36.366694 7753 tablet_copy_client.cc:538] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:36.370169 7762 tablet_copy_client.cc:538] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:36.377097 7753 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:36.379050 7762 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:36.396162 7762 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:36.396976 7762 tablet_bootstrap.cc:492] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:36.399451 7762 ts_tablet_manager.cc:1403] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.002s
I20260620 19:58:36.402318 7762 raft_consensus.cc:359] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.403612 7762 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:36.404297 7762 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.406752 7762 ts_tablet_manager.cc:1434] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260620 19:58:36.411569 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-dc6cbf763dbc4c3888feddf378ef4405 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:36.411952 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-dc6cbf763dbc4c3888feddf378ef4405 on tablet dc6cbf763dbc4c3888feddf378ef4405 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:36.415510 7753 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:36.421787 7753 tablet_bootstrap.cc:492] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:36.422495 7753 ts_tablet_manager.cc:1403] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.000s
I20260620 19:58:36.426535 7753 raft_consensus.cc:359] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.427220 7753 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:36.427807 7753 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:36.430725 7753 ts_tablet_manager.cc:1434] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:58:36.432497 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-ee78433ca73b4ea8b2b24535036f6369 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:36.432843 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-ee78433ca73b4ea8b2b24535036f6369 on tablet ee78433ca73b4ea8b2b24535036f6369 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:36.511291 6231 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:36.512379 7706 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:36.540666 6374 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:36.542104 7706 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:36.574082 6910 raft_consensus.cc:1217] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:36.593313 6301 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:36.594517 7733 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:36.624756 6231 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:36.626657 7733 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:36.670452 6910 raft_consensus.cc:1217] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:36.680405 6910 raft_consensus.cc:1217] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:36.789382 6231 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:36.790620 7645 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:36.803196 6907 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:36.807067 7774 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:36.887152 6907 raft_consensus.cc:1217] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:36.978086 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:37.029428 6375 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:37.029991 6375 raft_consensus.cc:606] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:37.032590 6306 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:37.033094 6306 raft_consensus.cc:606] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:37.034233 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:58:37.035984 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:37.036638 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:37.075551 7748 raft_consensus.cc:1064] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:37.078096 7748 consensus_queue.cc:237] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.087625 7645 raft_consensus.cc:1064] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:37.089815 7645 consensus_queue.cc:237] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.091073 6910 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.092535 7728 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:37.096781 6306 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.098222 6907 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.098385 6374 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.099692 7774 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:37.100111 7778 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:37.105603 7748 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:37.108434 6306 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.109786 7667 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:37.138008 6231 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.139338 7645 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:37.145262 7728 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.151185 6377 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.151685 7774 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.184079 6306 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.185390 6231 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.193485 6909 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.195896 7712 raft_consensus.cc:1064] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:37.203696 6130 catalog_manager.cc:5696] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:37.204188 6128 catalog_manager.cc:5696] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:37.231348 7712 raft_consensus.cc:1068] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: Unable to promote non-voter 20bd146dc0cb4917a1a5ce05ec609ce1: Service unavailable: leader transfer in progress
W20260620 19:58:37.218217 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 62079.9us
I20260620 19:58:37.249073 6301 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.272078 6909 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.299319 7645 raft_consensus.cc:993] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:37.299782 7645 raft_consensus.cc:1081] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:37.301810 6301 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52348
I20260620 19:58:37.302436 6301 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:37.303078 6301 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:37.314467 6301 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.316278 6301 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:37.317052 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:37.317575 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:37.317631 6377 raft_consensus.cc:3055] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:37.318154 6907 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:37.318261 6377 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:37.319661 6377 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.320861 6377 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:37.329839 6377 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:37.332823 6273 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:37.332986 6231 consensus_queue.cc:237] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.334286 7706 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:37.335636 7787 raft_consensus.cc:993] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:37.336062 7787 raft_consensus.cc:1081] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:37.336231 7728 raft_consensus.cc:1064] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:37.337172 6907 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:37.340059 6226 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:37.340085 7706 raft_consensus.cc:697] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:37.340571 6226 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:37.340960 6226 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:37.341331 7706 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.347443 6226 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:37.349754 6226 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:37.347559 7728 consensus_queue.cc:237] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.351187 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:37.351765 6377 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:37.357400 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:37.357963 6301 raft_consensus.cc:3055] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:37.358219 6301 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:37.358922 6301 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:37.386734 6301 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:37.387373 6306 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:37.390266 7650 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:37.397728 6301 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:37.411446 6375 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.398699 7748 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.413794 7650 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:37.401206 6909 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:37.403601 6306 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.407582 6377 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:37.408710 6907 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.416141 6375 consensus_queue.cc:237] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.421329 6195 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:37.421787 6907 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:37.422706 6301 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:37.423302 6307 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:37.423149 7667 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:37.424577 7645 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:37.426177 7778 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:37.432173 7645 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.442660 6307 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.448217 6907 raft_consensus.cc:2955] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.456813 7800 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260620 19:58:37.460990 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet e617e76f53974a7189d97fe9f8e69a8b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:37.454079 6128 catalog_manager.cc:5696] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:37.511267 6129 catalog_manager.cc:5696] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.516166 7778 raft_consensus.cc:697] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:37.517148 7778 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:37.520354 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 87954.8us
I20260620 19:58:37.522372 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet af9232a7f82e4c3f911b856592bd6ba9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:37.528879 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet af9232a7f82e4c3f911b856592bd6ba9 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:37.531345 7728 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:37.531222 7799 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:37.539790 7802 tablet_replica.cc:333] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:37.539559 6910 raft_consensus.cc:2955] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.542358 7802 raft_consensus.cc:2243] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:37.543519 7802 raft_consensus.cc:2272] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:37.544945 6128 catalog_manager.cc:5696] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.545913 7748 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.557065 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet e617e76f53974a7189d97fe9f8e69a8b with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:37.586802 7802 ts_tablet_manager.cc:1916] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:37.592283 6307 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.594882 6909 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.629954 6128 catalog_manager.cc:5696] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:37.631750 6129 catalog_manager.cc:5696] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.598286 6375 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.643411 7802 ts_tablet_manager.cc:1929] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:37.644073 7802 log.cc:1199] T af9232a7f82e4c3f911b856592bd6ba9 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/af9232a7f82e4c3f911b856592bd6ba9
I20260620 19:58:37.647924 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet af9232a7f82e4c3f911b856592bd6ba9 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:37.650578 7805 tablet_replica.cc:333] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:37.650486 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:37.651695 6231 raft_consensus.cc:606] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Received request to transfer leadership
I20260620 19:58:37.661285 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:37.663280 6231 raft_consensus.cc:606] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Received request to transfer leadership
I20260620 19:58:37.671314 6231 consensus_queue.cc:237] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:37.673609 6226 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:37.674175 6226 raft_consensus.cc:606] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Received request to transfer leadership
W20260620 19:58:37.678431 6178 auto_rebalancer.cc:855] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260620 19:58:37.678781 6178 auto_rebalancer.cc:863] Could not move replica: Remote error: Service unavailable: leadership transfer for ee78433ca73b4ea8b2b24535036f6369 already in progress
W20260620 19:58:37.679059 6178 auto_rebalancer.cc:301] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for ee78433ca73b4ea8b2b24535036f6369 already in progress
I20260620 19:58:37.680523 6375 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:37.678442 7805 raft_consensus.cc:2243] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:37.680546 6909 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:37.681999 7805 raft_consensus.cc:2272] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:37.682153 7778 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:37.683647 7798 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:37.686010 7805 ts_tablet_manager.cc:1916] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:37.690325 7797 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.691951 6909 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.696813 6378 raft_consensus.cc:2955] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.707697 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet dc6cbf763dbc4c3888feddf378ef4405 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:37.708487 7805 ts_tablet_manager.cc:1929] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:37.708900 7805 log.cc:1199] T e617e76f53974a7189d97fe9f8e69a8b P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/e617e76f53974a7189d97fe9f8e69a8b
I20260620 19:58:37.711041 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet e617e76f53974a7189d97fe9f8e69a8b (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:37.714002 6128 catalog_manager.cc:5696] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 3 to 4, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:37.727798 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet dc6cbf763dbc4c3888feddf378ef4405 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:37.732388 7807 tablet_replica.cc:333] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:37.733420 7807 raft_consensus.cc:2243] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:37.734253 7807 raft_consensus.cc:2272] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:37.738303 7807 ts_tablet_manager.cc:1916] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:37.752701 7807 ts_tablet_manager.cc:1929] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:37.753019 7807 log.cc:1199] T dc6cbf763dbc4c3888feddf378ef4405 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/dc6cbf763dbc4c3888feddf378ef4405
I20260620 19:58:37.754261 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet dc6cbf763dbc4c3888feddf378ef4405 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:37.923010 6375 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:37.924114 7786 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:37.935611 6909 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:37.940871 7786 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:38.013666 6909 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Advancing to term 2
I20260620 19:58:38.015947 6307 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:38.017843 7798 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:38.020088 6909 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:38.021389 7798 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:38.026328 7778 raft_consensus.cc:993] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:38.026805 7778 raft_consensus.cc:1081] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:38.032039 6307 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52350
I20260620 19:58:38.032562 6307 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:38.032889 6307 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:38.035995 6375 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:38.037186 7778 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:38.040560 7797 raft_consensus.cc:1064] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
W20260620 19:58:38.040933 7797 raft_consensus.cc:1068] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: Unable to promote non-voter 20bd146dc0cb4917a1a5ce05ec609ce1: Service unavailable: leader transfer in progress
I20260620 19:58:38.040647 6307 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:38.041704 6306 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:38.045873 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:38.046420 6226 raft_consensus.cc:3055] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:38.046772 6226 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:38.047521 6226 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:38.050328 7800 consensus_queue.cc:1175] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" responder_term: 2 status { last_received { term: 2 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 2 index: 3 } }
I20260620 19:58:38.048519 6226 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:38.072672 6226 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:38.073084 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:38.073673 6375 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:38.077986 6271 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:38.078787 7786 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:38.079439 7786 raft_consensus.cc:697] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:38.078871 6307 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:38.080261 7786 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:38.083258 6375 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:38.086083 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:38.091828 6129 catalog_manager.cc:5696] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:38.124125 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "098d4ada753442e1899d391a485095aa"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:38.124620 6307 raft_consensus.cc:606] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:38.126801 6307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:38.127275 6307 raft_consensus.cc:606] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:38.128255 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:58:38.129779 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:38.130309 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:38.356220 7706 raft_consensus.cc:993] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:38.356824 7786 raft_consensus.cc:1081] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:38.358057 6226 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:38.358493 6226 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:38.358832 6226 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:38.363093 6226 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:38.364396 6226 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:38.365010 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:38.365188 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:38.365499 6307 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:38.365793 6378 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:38.365829 6307 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:38.366549 6307 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:38.367451 6307 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:38.369967 6378 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:38.370877 6198 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:38.371397 6307 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:38.371446 7798 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:38.371968 7798 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:38.372673 7798 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:38.381249 6129 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:38.495910 6224 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:38.496886 7818 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:38.505234 6910 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Advancing to term 3
I20260620 19:58:38.505784 6375 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:38.508467 7818 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:38.517892 6910 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:38.519869 7786 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:38.528396 7786 raft_consensus.cc:1064] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:38.530856 7786 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:38.535758 6378 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:38.537120 6909 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:38.537134 6224 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:38.538489 7786 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:38.540139 7786 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:38.540860 7818 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:38.548557 7786 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.550542 6224 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.552956 6910 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.559768 6378 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.567811 6130 catalog_manager.cc:5696] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:38.584824 6306 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:38.590916 6909 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:38.590919 6378 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:38.592563 7786 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:38.593421 7706 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:38.599224 7786 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.600986 6378 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.608523 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet ee78433ca73b4ea8b2b24535036f6369 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:38.608912 6910 raft_consensus.cc:2955] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.615136 6130 catalog_manager.cc:5696] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 5 to 6, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:38.627935 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet ee78433ca73b4ea8b2b24535036f6369 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:38.639626 7821 tablet_replica.cc:333] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:38.640625 7821 raft_consensus.cc:2243] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:38.641475 7821 raft_consensus.cc:2272] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:38.644836 7821 ts_tablet_manager.cc:1916] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:38.655910 7821 ts_tablet_manager.cc:1929] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260620 19:58:38.656237 7821 log.cc:1199] T ee78433ca73b4ea8b2b24535036f6369 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/ee78433ca73b4ea8b2b24535036f6369
I20260620 19:58:38.657615 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet ee78433ca73b4ea8b2b24535036f6369 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:38.703629 7706 raft_consensus.cc:993] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:38.704177 7786 raft_consensus.cc:1081] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:38.705535 6378 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "098d4ada753442e1899d391a485095aa"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49014
I20260620 19:58:38.706076 6378 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:38.706450 6378 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:38.711571 6378 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:38.712991 6378 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:38.714136 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:38.714105 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:38.714848 6306 raft_consensus.cc:3055] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:38.714934 6224 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:38.715198 6306 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:38.715915 6306 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:38.716822 6306 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:38.721498 6224 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:38.722493 6306 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:38.722759 6345 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:38.724380 7645 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:38.724797 7645 raft_consensus.cc:697] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:38.725555 7645 consensus_queue.cc:237] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:38.733985 6129 catalog_manager.cc:5696] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 2, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:38.802948 6306 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:38.808043 6226 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:38.808046 6378 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 19:58:38.808202 6271 consensus_peers.cc:597] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48323077f2994fe383c12116b6e89657. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:38.809382 7786 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:38.810060 7818 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:38.817256 7818 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.819478 6378 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.819684 6226 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.835987 6129 catalog_manager.cc:5696] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.837592 6231 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
W20260620 19:58:38.880529 6196 consensus_peers.cc:597] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f202638d921b4ef2b75223ca54d70e04. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:38.880662 6301 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:38.881465 6307 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:38.882642 7828 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:38.882781 7829 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:38.882943 6375 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:58:38.884186 7828 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:38.902804 6378 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:38.904467 7829 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:38.913034 7778 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.926152 6301 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.927578 6224 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:38.934141 6374 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
W20260620 19:58:38.945706 6196 consensus_peers.cc:597] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cda425863f04d209a457c07ec3064d1. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:38.947247 6378 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:38.948915 7828 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:38.948861 6301 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:38.950886 6129 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 1 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:38.952231 7828 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:38.958364 7829 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.969797 6378 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.973246 6224 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:38.980276 6307 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:38.981493 6126 catalog_manager.cc:5696] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260620 19:58:38.993996 6196 consensus_peers.cc:597] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92d22c99dfc64cc1bb999795e691d1ad. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:38.995110 6301 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260620 19:58:38.996039 6374 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260620 19:58:38.996312 7828 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:38.997498 7829 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:39.011519 7778 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:39.013414 6307 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:39.020552 6374 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:39.030963 6126 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index -1 to 5, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:39.167629 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:39.186507 6231 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:39.187978 7774 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:39.198163 6301 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:39.202314 7774 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:39.234133 6378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:39.234742 6378 raft_consensus.cc:606] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:39.239935 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:39.240470 6231 raft_consensus.cc:606] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:39.241688 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:58:39.243141 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:39.243784 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:39.433339 7840 ts_tablet_manager.cc:933] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:39.435297 7840 tablet_copy_client.cc:323] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.130:42141
I20260620 19:58:39.436753 6317 tablet_copy_service.cc:140] P 39662d01be7a44f682d160227f457661: Received BeginTabletCopySession request for tablet 48323077f2994fe383c12116b6e89657 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:59724)
I20260620 19:58:39.437204 6317 tablet_copy_service.cc:161] P 39662d01be7a44f682d160227f457661: Beginning new tablet copy session on tablet 48323077f2994fe383c12116b6e89657 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:59724: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-48323077f2994fe383c12116b6e89657
I20260620 19:58:39.443936 6317 tablet_copy_source_session.cc:215] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:39.447304 7840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48323077f2994fe383c12116b6e89657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:39.451489 7842 ts_tablet_manager.cc:933] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:39.453457 7842 tablet_copy_client.cc:323] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:39.455004 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet f202638d921b4ef2b75223ca54d70e04 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:39.455503 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet f202638d921b4ef2b75223ca54d70e04 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-f202638d921b4ef2b75223ca54d70e04
I20260620 19:58:39.462431 6242 tablet_copy_source_session.cc:215] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:39.464891 7840 tablet_copy_client.cc:806] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:39.465500 7840 tablet_copy_client.cc:670] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:39.465520 7842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f202638d921b4ef2b75223ca54d70e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:39.469197 7840 tablet_copy_client.cc:538] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:39.480916 7840 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:39.497951 7844 ts_tablet_manager.cc:933] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:39.501048 7842 tablet_copy_client.cc:806] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:39.502094 7842 tablet_copy_client.cc:670] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:39.518317 7844 tablet_copy_client.cc:323] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:39.520273 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 6cda425863f04d209a457c07ec3064d1 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:39.520803 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 6cda425863f04d209a457c07ec3064d1 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-6cda425863f04d209a457c07ec3064d1
I20260620 19:58:39.530704 6242 tablet_copy_source_session.cc:215] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:39.533964 7844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cda425863f04d209a457c07ec3064d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:39.536022 7842 tablet_copy_client.cc:538] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:39.551481 7847 ts_tablet_manager.cc:933] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:39.557461 7798 raft_consensus.cc:993] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: : Instructing follower 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:39.557986 7798 raft_consensus.cc:1081] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Signalling peer 39662d01be7a44f682d160227f457661 to start an election
I20260620 19:58:39.559861 6301 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04"
dest_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:52350
I20260620 19:58:39.561612 7847 tablet_copy_client.cc:323] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.129:36725
I20260620 19:58:39.560576 6301 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:39.688858 6301 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:39.718724 7844 tablet_copy_client.cc:806] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:39.719373 7844 tablet_copy_client.cc:670] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:39.735363 7774 raft_consensus.cc:993] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: : Instructing follower 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:39.737886 7838 raft_consensus.cc:1081] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Signalling peer 20bd146dc0cb4917a1a5ce05ec609ce1 to start an election
I20260620 19:58:39.752266 7842 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:39.753177 6909 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:53406
I20260620 19:58:39.754568 6242 tablet_copy_service.cc:140] P eb2f228ebe40440d877be6302bba2cd6: Received BeginTabletCopySession request for tablet 92d22c99dfc64cc1bb999795e691d1ad from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:35548)
I20260620 19:58:39.790692 7844 tablet_copy_client.cc:538] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:39.810580 6909 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:39.814489 6909 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:39.810910 6242 tablet_copy_service.cc:161] P eb2f228ebe40440d877be6302bba2cd6: Beginning new tablet copy session on tablet 92d22c99dfc64cc1bb999795e691d1ad from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:35548: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-92d22c99dfc64cc1bb999795e691d1ad
I20260620 19:58:39.819764 7840 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:39.820701 7840 tablet_bootstrap.cc:492] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:39.821365 7840 ts_tablet_manager.cc:1403] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.341s user 0.027s sys 0.057s
I20260620 19:58:39.826277 6909 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:39.828758 7840 raft_consensus.cc:359] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.833840 7840 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:39.834547 7840 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.844610 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:39.845170 6224 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:39.845567 7840 ts_tablet_manager.cc:1434] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260620 19:58:39.793481 6301 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.851368 6317 tablet_copy_service.cc:342] P 39662d01be7a44f682d160227f457661: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-48323077f2994fe383c12116b6e89657 received from {username='slave'} at 127.0.0.1:59724
I20260620 19:58:39.851883 6317 tablet_copy_service.cc:434] P 39662d01be7a44f682d160227f457661: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-48323077f2994fe383c12116b6e89657 on tablet 48323077f2994fe383c12116b6e89657 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:39.852059 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:39.852612 6378 raft_consensus.cc:3055] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:39.852869 6378 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:39.854948 6378 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:39.855922 6378 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:39.859922 6909 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:39.829547 7844 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:39.875195 6224 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:39.876788 6875 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:39.877645 7853 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:39.878193 7853 raft_consensus.cc:697] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:39.879288 7853 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:39.864986 6242 tablet_copy_source_session.cc:215] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:58:39.898307 6301 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:39.899219 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:39.910089 6374 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:39.912180 6129 catalog_manager.cc:5696] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:39.900027 7847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92d22c99dfc64cc1bb999795e691d1ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:58:39.917483 6374 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:39.918581 6273 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:39.919276 7786 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:39.919692 7786 raft_consensus.cc:697] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:39.920467 7786 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.927891 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:39.928454 6231 raft_consensus.cc:3055] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:39.928707 6231 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:39.929333 6231 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.935597 6231 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:39.935794 6126 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:39.901990 7844 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:39.940346 7844 tablet_bootstrap.cc:492] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:39.942246 7844 ts_tablet_manager.cc:1403] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.112s user 0.022s sys 0.013s
I20260620 19:58:39.947815 7844 raft_consensus.cc:359] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.948618 7844 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:39.890273 6378 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:39.949226 7847 tablet_copy_client.cc:806] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:58:39.954299 7847 tablet_copy_client.cc:670] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:58:39.955646 7844 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.960541 7847 tablet_copy_client.cc:538] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:58:39.968204 7844 ts_tablet_manager.cc:1434] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20260620 19:58:39.969844 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-6cda425863f04d209a457c07ec3064d1 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:39.970477 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-6cda425863f04d209a457c07ec3064d1 on tablet 6cda425863f04d209a457c07ec3064d1 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:39.978017 6231 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:39.982221 7847 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:58:39.992632 7842 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:39.993347 7842 tablet_bootstrap.cc:492] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:39.993927 7842 ts_tablet_manager.cc:1403] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.242s user 0.038s sys 0.083s
I20260620 19:58:39.997284 7842 raft_consensus.cc:359] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:39.998064 7842 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:58:39.998693 7842 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:40.005013 7842 ts_tablet_manager.cc:1434] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260620 19:58:40.007071 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-f202638d921b4ef2b75223ca54d70e04 received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:40.007452 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-f202638d921b4ef2b75223ca54d70e04 on tablet f202638d921b4ef2b75223ca54d70e04 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:40.027529 6301 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:40.028124 7847 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:58:40.028163 6301 raft_consensus.cc:606] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership
I20260620 19:58:40.029168 7847 tablet_bootstrap.cc:492] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:58:40.029824 7847 ts_tablet_manager.cc:1403] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 0.048s user 0.024s sys 0.010s
I20260620 19:58:40.031932 6301 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:40.032357 6301 raft_consensus.cc:606] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership
I20260620 19:58:40.032223 7847 raft_consensus.cc:359] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:40.033068 7847 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
W20260620 19:58:40.033347 6114 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for f202638d921b4ef2b75223ca54d70e04 already in progress [suppressed 1 similar messages]
I20260620 19:58:40.033703 7847 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:40.036383 7847 ts_tablet_manager.cc:1434] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:58:40.037508 6301 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:40.037770 6242 tablet_copy_service.cc:342] P eb2f228ebe40440d877be6302bba2cd6: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-92d22c99dfc64cc1bb999795e691d1ad received from {username='slave'} at 127.0.0.1:35548
I20260620 19:58:40.038019 6301 raft_consensus.cc:606] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership
I20260620 19:58:40.038208 6242 tablet_copy_service.cc:434] P eb2f228ebe40440d877be6302bba2cd6: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-92d22c99dfc64cc1bb999795e691d1ad on tablet 92d22c99dfc64cc1bb999795e691d1ad with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:40.041154 6910 raft_consensus.cc:1217] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20260620 19:58:40.048370 6178 auto_rebalancer.cc:855] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260620 19:58:40.048718 6178 auto_rebalancer.cc:863] Could not move replica: Remote error: Service unavailable: leadership transfer for f202638d921b4ef2b75223ca54d70e04 already in progress
W20260620 19:58:40.048978 6178 auto_rebalancer.cc:301] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for f202638d921b4ef2b75223ca54d70e04 already in progress
I20260620 19:58:40.246402 6910 raft_consensus.cc:1217] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260620 19:58:40.298352 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:40.332145 6910 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Advancing to term 2
I20260620 19:58:40.332455 6374 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:40.335567 7853 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260620 19:58:40.340325 6910 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:40.341585 7851 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:40.345139 6374 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:40.347087 7851 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:40.371640 7852 raft_consensus.cc:1064] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
W20260620 19:58:40.372056 7852 raft_consensus.cc:1068] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: Unable to promote non-voter 20bd146dc0cb4917a1a5ce05ec609ce1: Service unavailable: leader transfer in progress
I20260620 19:58:40.389807 7852 raft_consensus.cc:993] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: : Instructing follower 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:40.390288 7851 raft_consensus.cc:1081] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Signalling peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 to start an election
I20260620 19:58:40.391883 6224 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 19:58:40.393340 7786 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:40.394362 6377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:49014
I20260620 19:58:40.420768 6301 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79a07a8804784405920baab70a535338"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:58:40.421346 6301 raft_consensus.cc:606] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:40.422438 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260620 19:58:40.423879 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:40.424614 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:40.427954 6377 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:40.428489 6377 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:40.435253 6377 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:40.448673 6377 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:40.460564 6910 raft_consensus.cc:1217] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 19:58:40.465669 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:40.468655 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:40.468240 6231 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:40.471746 7853 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:40.507934 6306 raft_consensus.cc:3055] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:40.508327 6306 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:40.509136 6306 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:40.510521 6306 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:40.522410 6306 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:40.528256 6344 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:40.529338 7645 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:40.529788 7645 raft_consensus.cc:697] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:40.530586 7645 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:40.543802 6129 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:40.624064 7850 raft_consensus.cc:1064] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:40.625948 7850 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:40.631703 6910 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:40.631968 6375 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:40.633446 7850 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:40.634913 6306 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:40.635134 7854 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:40.636675 7829 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:40.645056 7854 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.648198 6306 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.712420 6129 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 5 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.719743 6910 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.729411 6375 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.771492 6224 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:40.775991 6910 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 19:58:40.778955 6377 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 19:58:40.780767 7854 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:40.783136 7854 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:40.789911 7850 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.791425 6377 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.793334 6909 raft_consensus.cc:2955] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.809592 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 92d22c99dfc64cc1bb999795e691d1ad with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:40.809144 6126 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: config changed from index 6 to 7, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:40.830884 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet 92d22c99dfc64cc1bb999795e691d1ad with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:40.860762 7883 tablet_replica.cc:333] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:40.862002 7883 raft_consensus.cc:2243] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:40.863274 7883 raft_consensus.cc:2272] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:40.863903 7786 raft_consensus.cc:993] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: : Instructing follower eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:40.864388 7870 raft_consensus.cc:1081] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Signalling peer eb2f228ebe40440d877be6302bba2cd6 to start an election
I20260620 19:58:40.865942 6231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "79a07a8804784405920baab70a535338"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:45808
I20260620 19:58:40.866613 6231 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:58:40.866935 6231 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:40.885617 7852 raft_consensus.cc:1064] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:40.888139 7852 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:40.869879 7883 ts_tablet_manager.cc:1916] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:40.902221 6375 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:40.904987 6909 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:40.906757 7851 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:40.910384 7786 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:40.941390 6231 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:40.947615 6231 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:40.949118 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:40.949353 6231 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 19:58:40.949646 6306 raft_consensus.cc:3055] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:40.949959 6306 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:40.950920 6306 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:40.953135 7786 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:40.964401 6306 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:40.975376 6306 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:58:40.992051 7888 log.cc:927] Time spent T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260620 19:58:41.011475 7870 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.013624 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:41.014221 6907 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:41.020845 6907 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:41.024549 6307 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:41.025162 6196 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:41.025766 6910 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.027127 7892 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:41.027652 7892 raft_consensus.cc:697] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:41.028470 7892 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:41.032776 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 47149.8us
I20260620 19:58:41.033309 7645 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:41.036841 7883 ts_tablet_manager.cc:1929] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20260620 19:58:41.044077 7883 log.cc:1199] T 92d22c99dfc64cc1bb999795e691d1ad P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/92d22c99dfc64cc1bb999795e691d1ad
I20260620 19:58:41.045856 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet 92d22c99dfc64cc1bb999795e691d1ad (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:41.046972 6377 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.051605 6129 catalog_manager.cc:5696] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:41.074337 7854 raft_consensus.cc:1064] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:41.075687 6224 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:41.076467 6129 catalog_manager.cc:5696] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.078903 6907 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Advancing to term 3
I20260620 19:58:41.082326 6224 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:41.084592 7854 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:41.096726 6224 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.083961 7884 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260620 19:58:41.098739 6306 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:41.129798 6907 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260620 19:58:41.131359 7645 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:41.134378 6375 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:41.140015 6910 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEARNER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:41.141381 7889 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:41.141649 7851 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:41.149507 6909 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260620 19:58:41.151225 6374 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:41.153676 7854 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:41.162490 7786 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.163388 6307 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 19:58:41.174660 7852 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:41.175565 6114 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 48323077f2994fe383c12116b6e89657 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:41.180163 6909 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.186022 6129 catalog_manager.cc:5696] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 reported cstate change: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New cstate: current_term: 1 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:41.197250 6375 raft_consensus.cc:2955] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.205302 7889 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:41.208820 7838 raft_consensus.cc:1064] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:58:41.211340 7838 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:41.219291 6307 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:41.220712 7645 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:41.226898 6222 tablet_service.cc:1558] Processing DeleteTablet for tablet 48323077f2994fe383c12116b6e89657 with delete_type TABLET_DATA_TOMBSTONED (TS eb2f228ebe40440d877be6302bba2cd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45792
I20260620 19:58:41.232074 6224 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:41.244151 7903 tablet_replica.cc:333] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: stopping tablet replica
I20260620 19:58:41.245141 7838 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:41.246469 6907 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEARNER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:41.248637 7889 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.250516 7903 raft_consensus.cc:2243] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:41.250324 6374 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.251395 7838 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:41.252883 7903 raft_consensus.cc:2272] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:41.259444 7903 ts_tablet_manager.cc:1916] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:41.261873 7884 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.271544 6910 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.278434 7903 ts_tablet_manager.cc:1929] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260620 19:58:41.279207 7903 log.cc:1199] T 48323077f2994fe383c12116b6e89657 P eb2f228ebe40440d877be6302bba2cd6: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-0-root/wals/48323077f2994fe383c12116b6e89657
I20260620 19:58:41.280848 6115 catalog_manager.cc:5027] TS eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): tablet 48323077f2994fe383c12116b6e89657 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:41.288502 6909 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.277791 6126 catalog_manager.cc:5696] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:41.298044 6231 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.304124 6306 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.320686 6129 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 2 to 5, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:41.331140 6307 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.341481 6231 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:41.348938 6910 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:41.350287 7829 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:41.353366 6307 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:41.354671 6377 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:41.354925 7892 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:41.356211 7889 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.364367 6909 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.366210 6307 raft_consensus.cc:2955] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.367820 6907 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:41.369477 6231 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:41.369390 7645 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:41.370880 7904 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:41.380146 6115 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet 6cda425863f04d209a457c07ec3064d1 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:41.390743 7645 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.404840 6117 catalog_manager.cc:5209] ChangeConfig:REMOVE_PEER RPC for tablet f202638d921b4ef2b75223ca54d70e04 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 19:58:41.404718 6224 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.405341 6129 catalog_manager.cc:5696] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index 4 to 5, VOTER 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) evicted. New cstate: current_term: 2 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:41.406088 6909 raft_consensus.cc:2955] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:58:41.415659 6306 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:41.419036 7889 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:41.423933 6128 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: config changed from index 5 to 6, VOTER 39662d01be7a44f682d160227f457661 (127.4.244.130) evicted. New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:41.434672 6368 tablet_service.cc:1558] Processing DeleteTablet for tablet 6cda425863f04d209a457c07ec3064d1 with delete_type TABLET_DATA_TOMBSTONED (TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49006
I20260620 19:58:41.440486 6909 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:41.442245 6293 tablet_service.cc:1558] Processing DeleteTablet for tablet f202638d921b4ef2b75223ca54d70e04 with delete_type TABLET_DATA_TOMBSTONED (TS 39662d01be7a44f682d160227f457661 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52340
I20260620 19:58:41.443186 7883 tablet_replica.cc:333] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: stopping tablet replica
I20260620 19:58:41.444164 7883 raft_consensus.cc:2243] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:41.444563 7906 tablet_replica.cc:333] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: stopping tablet replica
I20260620 19:58:41.444810 7883 raft_consensus.cc:2272] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:41.446022 7906 raft_consensus.cc:2243] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 19:58:41.446815 7906 raft_consensus.cc:2272] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 19:58:41.455449 7906 ts_tablet_manager.cc:1916] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:41.441668 7889 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260620 19:58:41.458839 7883 ts_tablet_manager.cc:1916] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 19:58:41.480121 7883 ts_tablet_manager.cc:1929] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260620 19:58:41.480585 7883 log.cc:1199] T f202638d921b4ef2b75223ca54d70e04 P 39662d01be7a44f682d160227f457661: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-1-root/wals/f202638d921b4ef2b75223ca54d70e04
I20260620 19:58:41.482326 6114 catalog_manager.cc:5027] TS 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): tablet f202638d921b4ef2b75223ca54d70e04 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:41.494277 7906 ts_tablet_manager.cc:1929] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260620 19:58:41.494760 7906 log.cc:1199] T 6cda425863f04d209a457c07ec3064d1 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Deleting WAL directory at /tmp/dist-test-taskQFzxbe/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1781985411095902-5074-0/minicluster-data/ts-2-root/wals/6cda425863f04d209a457c07ec3064d1
I20260620 19:58:41.496364 6117 catalog_manager.cc:5027] TS 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): tablet 6cda425863f04d209a457c07ec3064d1 (table test-workload [id=bdc4fe719f514b06a7419a6a405069a5]) successfully deleted
I20260620 19:58:41.512054 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:41.545706 6231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92212144534e4394844003016ba67a47"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:41.546207 6231 raft_consensus.cc:606] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:41.548297 6374 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:41.548710 6374 raft_consensus.cc:606] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:41.549659 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:58:41.550590 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:41.551108 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:42.813478 7910 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:42.814198 7910 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:42.816943 7910 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:42.990129 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260620 19:58:42.991248 6161 debug-util.cc:398] Leaking SignalData structure 0x7b080031bd80 after lost signal to thread 5077
I20260620 19:58:43.421802 7910 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:43.421803 7909 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:43.423334 7910 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:43.423357 7909 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.427054 7909 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.427227 7910 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:43.427955 7910 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:43.428858 7910 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.431788 7910 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.434863 7910 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:43.427702 7909 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:43.574468 7909 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:43.577703 7909 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:43.578387 7909 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:43.579346 7909 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:43.582020 7909 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:43.587867 7910 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:43.497853 7913 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:43.661211 7914 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:58:43.681010 6161 debug-util.cc:398] Leaking SignalData structure 0x7b080031c780 after lost signal to thread 6164
W20260620 19:58:43.781559 6272 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.001s sys 0.011s
W20260620 19:58:43.784862 6272 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.001s sys 0.011s
I20260620 19:58:43.787406 7913 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.791066 7913 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.792641 7913 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:43.793658 7913 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.786710 7910 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:43.799396 7910 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:43.800289 7910 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.803061 7910 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.797500 7914 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:43.803539 7909 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:43.805650 7914 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:43.806234 7914 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:43.797423 7911 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:43.798750 7912 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:43.809703 7910 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:43.810478 7911 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.810904 7910 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.810860 7912 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.813179 7915 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:43.814025 7911 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.816286 7912 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.806435 7909 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:43.807104 7914 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.101794 7913 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.846575 7645 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.108757 7909 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:43.889035 7873 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.184373 7919 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.185340 7919 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.188237 7919 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.193322 7910 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:44.195263 6270 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.047s user 0.016s sys 0.005s
W20260620 19:58:44.202793 6270 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.055s user 0.017s sys 0.005s
I20260620 19:58:44.203270 7911 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.195566 7914 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.215503 7645 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.217617 7733 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.218874 7911 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.221130 7733 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.222898 7911 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.181856 7915 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.223717 7919 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.231245 7873 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.234704 7919 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.235801 7912 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.236277 7645 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.236524 7873 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.236418 7912 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.236977 7919 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.244628 7918 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.244637 7645 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.245787 7645 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.245689 7918 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.288861 7918 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.244925 7921 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.303785 7919 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.304307 7733 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.289405 7911 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.307446 7912 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.368147 7910 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.368467 7922 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.408850 7912 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.459574 7912 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.407819 7921 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.608583 7910 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.611433 7910 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.612110 7910 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.612895 7910 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.615857 7919 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.405515 7918 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.462565 7912 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.622917 7919 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.624181 7922 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.648087 7924 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.407845 7911 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.654585 7645 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.665366 7733 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.665613 7873 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.666275 7733 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.666675 7873 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.668917 7733 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.669541 7733 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.669612 7873 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.670288 7733 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.674762 7733 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.681483 7930 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.682368 7930 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.685074 7932 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.685402 7930 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.690164 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.695055 7911 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.699718 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.700860 6306 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:44.705744 7929 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.706761 7929 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.710461 7928 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.711225 7733 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.711338 7928 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.712077 7733 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.720396 7927 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.721310 7923 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.722636 7923 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.723433 7873 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.714394 7928 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.728317 7918 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.714352 7733 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.648890 7924 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.686034 7932 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:44.744097 7733 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.744832 7733 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.747341 7733 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.748369 7911 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.749169 7911 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.751770 7911 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.748044 7733 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.755267 7733 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.757675 7733 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.724347 7873 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.759641 7911 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.764065 7918 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.766258 7918 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.767030 7918 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.770115 7918 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.730079 7915 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.773979 7733 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.809396 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.810159 6306 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:44.781684 7929 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.779610 7923 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.824661 7943 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.795014 7930 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.827229 7733 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.828737 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.818151 7910 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.816336 7927 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.840278 7938 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.840714 7733 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.840826 7873 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.841349 7928 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.842355 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.843909 7924 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.845098 7922 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.846763 6306 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:44.847187 7935 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.823397 7911 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.841462 7873 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.856794 7909 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.857016 7940 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.857502 7934 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.857337 7928 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.831260 7929 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.851142 6301 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:44.835005 7923 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.825345 7943 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.841002 7938 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.850243 7930 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.878868 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.878520 7933 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.880003 7923 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.880554 7911 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.878429 7939 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.887168 7911 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.878409 7929 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.876140 7912 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.892635 7936 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.887753 7911 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.867120 7873 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.875844 7919 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.880856 7924 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.880262 7934 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.887501 7939 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.902838 7927 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.903913 7911 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.904163 7929 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.904325 7914 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.907213 7948 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.907621 7914 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.907274 7911 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.874850 7940 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.888329 7932 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:44.878543 7909 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.912204 7926 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:44.914499 7645 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.915818 7945 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.916318 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.901158 7910 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.917907 7645 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.922572 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.923259 7909 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.919659 7947 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.921609 7929 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:44.925357 7929 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.927438 7929 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:44.927116 7919 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.927920 7924 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.929256 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.929816 6307 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:44.930015 7924 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.931330 7928 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.931790 7873 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.908043 7948 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.929894 7919 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.916427 7945 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.919222 7938 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.920195 7930 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.924340 7947 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.933140 7927 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.913069 7936 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.952071 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.952672 7733 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.939790 7911 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.937819 7912 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.890430 7935 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.941008 7910 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:44.977340 7733 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.956912 7927 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.956094 7926 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.983124 7916 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:44.983989 7916 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.917141 7933 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:44.957645 7929 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.029031 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.981544 7936 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.975696 7914 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.021478 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.029670 6305 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:44.976835 7873 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.037104 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:44.975936 7928 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.040627 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.024953 7943 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:44.985460 7926 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.041491 6305 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:45.030309 7946 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:44.986919 7916 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:44.990370 7931 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.037374 7914 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.049325 7929 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.058863 7942 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.059670 7951 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.040244 7940 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.055357 7873 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.063869 7923 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.046914 7946 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.085839 7947 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.063889 7928 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.071930 7939 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.083868 7945 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.087598 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.086380 7934 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.100430 6909 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:45.068342 7914 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.080032 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.103550 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.086297 7910 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.100979 7911 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.100803 7952 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:45.080171 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.108229 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.114059 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.107242 7948 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.109169 7953 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.117336 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.118738 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.109591 7939 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.102541 7951 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.121335 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.084980 7931 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.122385 6377 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.122874 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.116701 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.124802 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.116580 7952 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.118250 6903 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:45.125869 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.126061 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.118036 7953 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.126727 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.118391 7947 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:45.102710 7942 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:45.112943 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 104248us
I20260620 19:58:45.143002 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.138582 6378 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.137737 6374 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:45.154847 6909 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:45.156980 7893 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.123616 7935 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.157275 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.189208 7929 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.187556 6381 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.191627 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.200330 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.206039 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.156510 7953 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.206524 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.206192 7955 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.206547 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.209676 7931 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.210094 7958 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:45.167711 6906 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:45.143002 7947 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.211793 6381 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:45.210942 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.190251 7954 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.208225 7937 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.148963 7939 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.216631 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.217125 7933 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.217305 7952 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.217557 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.217974 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.211035 7946 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.219949 6907 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:45.221956 7955 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.257085 7947 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.257681 7947 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:45.210522 7893 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.259241 7947 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.259903 7955 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.259985 6226 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:45.260175 7873 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.260752 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.208333 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.218218 7958 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.232625 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.258481 7928 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.254988 7954 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.250804 6379 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:45.293778 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.410372 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.431136 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.432550 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.289032 7955 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.438295 6907 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:45.409745 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.287830 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.295938 7914 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.294876 7939 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.471673 6227 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.367575 7893 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.284845 7942 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.469592 7955 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.295137 7946 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.289805 7937 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.456637 6905 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:45.515020 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.515547 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.531219 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.531725 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.516418 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.529889 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.532429 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.532572 6378 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.532863 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.533435 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.533943 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.484254 6372 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.534214 6378 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.535319 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.535627 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.517321 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.405679 7958 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.407490 7957 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.410578 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.517768 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:45.525553 6161 debug-util.cc:398] Leaking SignalData structure 0x7b08000f35a0 after lost signal to thread 6256
I20260620 19:58:45.512735 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.527946 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.543033 6298 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.528188 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.514789 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.528842 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.522243 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.544111 6907 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:45.531859 7954 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.533504 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.517077 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.515753 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.546962 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.547115 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.547562 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.548087 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.548331 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.548599 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.548858 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.549582 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.516600 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.530369 6302 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:45.528245 7947 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.536018 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.537007 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.540424 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.540897 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.535952 6372 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:45.543923 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.553793 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.554359 6378 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.532863 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.551258 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.552762 6305 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.550472 6303 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.534157 7946 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:45.558673 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.555697 7955 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.558190 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.559765 6232 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.556689 6306 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:45.559257 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.555655 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.563898 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.568202 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.568822 6226 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.563072 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.570499 6304 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:45.563382 6379 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.573589 6874 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.574435 7940 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.574574 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.574846 7940 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.575641 7940 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.579782 7937 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.579609 7957 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.580503 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.581099 6902 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:45.583575 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.578450 6874 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.592756 6273 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:45.593938 7851 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.594372 7851 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.594640 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.595383 6230 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.595945 6273 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.596470 7645 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.596794 7933 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.596858 6273 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.597566 7933 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.598050 7933 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.598296 6273 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.594820 7851 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.609012 7933 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.611809 7933 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.612547 7933 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:45.612982 7933 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.613435 7933 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:45.619870 7951 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.626334 6344 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.632393 7645 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:45.632786 7645 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.633129 7645 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:45.633503 7927 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.627887 7912 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.634902 7912 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.635313 7912 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.635722 7912 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.637519 7921 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.641640 7933 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.642958 6344 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:58:45.643793 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 65412.2us
I20260620 19:58:45.649092 7645 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.644500 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.591281 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.581678 7957 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:45.647034 6161 debug-util.cc:398] Leaking SignalData structure 0x7b08001f4160 after lost signal to thread 6331
I20260620 19:58:45.657112 6874 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.658509 7933 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.660215 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.660876 6232 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.663098 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.663862 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.664093 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.664479 6224 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:45.665010 6228 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:45.665438 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.663057 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.666184 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.666756 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.667121 6229 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:45.668396 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.669147 6227 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.666306 6231 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:45.670063 7733 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.658496 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.672801 7733 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.672994 6874 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.673378 7733 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.674253 7733 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.684602 7960 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.685221 7960 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.647816 7927 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260620 19:58:45.690241 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 47376.3us
I20260620 19:58:45.691134 6344 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.693099 7645 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.631217 7851 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:45.688961 6161 debug-util.cc:398] Leaking SignalData structure 0x7b0800300fc0 after lost signal to thread 6405
I20260620 19:58:45.648145 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.697297 7941 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.698690 7941 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:58:45.698796 6161 debug-util.cc:398] Leaking SignalData structure 0x7b0800309c80 after lost signal to thread 6935
I20260620 19:58:45.699082 7941 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.695493 7921 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.699645 7921 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.700058 7921 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.703022 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.703249 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.697328 7937 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.704178 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.704023 7937 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.705263 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.705339 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.705977 7937 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.694061 7957 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.706470 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.706507 7941 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.706954 7957 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.707309 6380 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:45.708130 6877 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.708278 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.708626 7941 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.709000 7957 leader_election.cc:290] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.709343 6877 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:45.710285 7917 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.710518 7928 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.711596 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.713295 6877 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.688782 7873 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.697175 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:45.716128 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 117904us
I20260620 19:58:45.714953 6344 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.717890 6225 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:45.717684 7948 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:45.692739 7927 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.720657 6877 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.721647 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.710940 7928 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:45.720201 6273 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.723654 6273 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.721529 7961 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.724824 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.724740 7961 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.725306 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.726754 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.727288 7838 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:45.719465 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.728643 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.727802 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.730455 6877 leader_election.cc:304] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.730762 6273 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.730674 6231 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:58:45.727658 7838 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.719805 7873 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.732412 6345 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:45.732167 6344 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:45.740600 7838 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:45.741467 7935 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.720769 7946 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.714778 7912 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.728266 7948 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.728989 7911 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.810286 7948 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.810739 7948 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.780441 7838 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:45.723241 7927 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:45.813328 7838 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.723353 7928 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.728812 7961 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.734557 7933 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260620 19:58:45.813867 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 138324us
I20260620 19:58:45.814291 7933 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.814581 6271 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.814976 7933 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.767112 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.770012 6344 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:58:45.817005 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 47050.6us
I20260620 19:58:45.817456 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.817622 6344 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.818572 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.819051 7964 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.819599 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.742801 7873 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.821583 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.822557 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.823184 7933 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.772642 7884 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.773102 7851 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.735754 6903 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:45.824024 7884 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:45.811609 7912 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.824451 7884 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.743866 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.824836 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.825915 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.775985 6345 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.779453 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:45.828054 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 52137us
I20260620 19:58:45.828940 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.780107 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.829411 7941 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:45.780668 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.829787 7913 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:45.830168 7913 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.824851 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.830746 7913 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:45.830806 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.831001 6903 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:45.831390 6231 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.832245 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.832522 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.832288 7884 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.833400 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.833890 7918 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.834316 7918 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.834424 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.834738 7918 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.835584 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.836262 6231 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.836990 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.829777 7941 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.838100 7941 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:45.781409 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.823987 7911 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.782306 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.838968 7911 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.782768 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.787240 7960 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.787271 7940 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.789511 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.790210 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.841636 6227 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:45.842339 7940 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:45.838001 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.790269 7963 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.843166 7963 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.843271 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.791062 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.843084 6231 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.843568 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.792246 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.792542 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.843565 7873 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.793520 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.793823 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.794112 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.846233 7851 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.846433 6224 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:45.846661 7851 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.847035 6303 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:45.847244 7965 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:45.847787 7965 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.847833 6271 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.848255 7965 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:45.837508 7913 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.848747 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.850113 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.846963 6345 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:45.850746 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.851619 6345 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.852092 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.852276 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.852422 7852 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.852916 7852 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.797560 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.853125 7925 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:45.853489 7852 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.853897 7925 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.854139 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:45.855232 6905 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:45.855353 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.856422 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:45.799985 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.800230 6874 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.800518 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.800640 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.743825 6877 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.801242 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.801697 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:45.843657 7963 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:58:45.859349 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 59198.4us
I20260620 19:58:45.802872 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:45.806079 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.806385 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.860694 6305 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:58:45.859362 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 115625us
I20260620 19:58:45.860924 6874 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:45.808024 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.792933 7733 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.721016 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.863015 6907 raft_consensus.cc:1240] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:45.863126 7917 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:45.863306 7935 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.863513 7917 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.863752 7935 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.863953 7917 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:45.864162 7935 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.820814 7914 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.823520 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.770977 7915 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.781391 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:45.840359 6226 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:45.847102 7851 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.869750 7915 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.870930 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.871181 7915 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.871496 6301 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:45.852864 6345 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:45.798262 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.854303 7925 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.798987 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.795939 7921 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.806118 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.861445 6877 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.875744 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.874459 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.864956 6271 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.865935 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.876940 6345 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:45.870314 6226 raft_consensus.cc:606] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:45.873351 6304 raft_consensus.cc:1240] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:45.866705 7914 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.820752 7920 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.884331 7733 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.877192 6305 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:45.881031 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.879153 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.866256 7929 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:45.883190 6271 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.865589 7913 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.846429 7873 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:45.889977 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.890422 7929 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:45.894860 7929 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.895236 7733 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.873975 7838 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.893528 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:45.894063 7913 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.893983 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.892808 6303 raft_consensus.cc:1240] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:45.896275 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:45.897436 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.862092 7918 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.884395 7914 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.898262 6301 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:45.875502 7925 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.900681 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.898489 7884 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.901328 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.904692 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.881841 7912 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.905107 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:45.910845 6302 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.866864 7928 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.884579 7920 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:45.888067 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:45.904064 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.914628 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:45.916724 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 55326.2us
I20260620 19:58:45.922909 6374 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:45.923013 6877 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.923669 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.925971 6196 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:45.926664 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 56228.8us
I20260620 19:58:45.926833 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:45.928514 7962 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.928965 7962 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:45.929399 7962 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:45.929065 7852 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:45.909490 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:45.922430 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:45.932641 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 55754.2us
W20260620 19:58:45.933236 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 50137.2us
I20260620 19:58:45.930073 7921 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:45.933915 7889 consensus_queue.cc:1059] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:45.966192 6198 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:45.921960 7964 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:45.937448 6271 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:45.934671 7946 consensus_queue.cc:1059] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:45.922367 7920 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:45.936249 6345 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:45.932152 7931 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:45.928536 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:58:45.995324 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 59152.9us
I20260620 19:58:45.930968 6298 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:45.914690 7940 consensus_queue.cc:1059] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260620 19:58:46.008747 7940 raft_consensus.cc:3055] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.009066 7940 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:45.931407 7933 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.011610 7852 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.003265 6345 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:46.004202 7884 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.004460 7889 raft_consensus.cc:3055] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.004944 7946 raft_consensus.cc:3055] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
W20260620 19:58:46.005496 6344 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.753s (SENT)
W20260620 19:58:46.007251 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 103878us
I20260620 19:58:46.007718 7961 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.002044 6875 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.009913 7940 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.020653 7884 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:46.024139 7889 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.023757 7921 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.002624 7931 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:46.045125 7884 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.040046 7961 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:46.060758 7961 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.045857 7873 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.058734 7921 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:46.059229 7931 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.033020 7946 raft_consensus.cc:740] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.069025 7946 consensus_queue.cc:260] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.067227 7921 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.078238 7889 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.083600 7940 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.085295 6198 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:58:46.086313 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 218256us
W20260620 19:58:46.087153 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 70467.6us
I20260620 19:58:46.089311 6875 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.097555 6198 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.098558 7959 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.090181 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:46.101431 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.090492 7889 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.090777 7946 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.109273 6198 leader_election.cc:304] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.102046 7873 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.100073 7960 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:58:46.108107 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.101949 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.111917 7928 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:46.119266 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 181836us
I20260620 19:58:46.120999 6198 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.102847 7911 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.114522 7918 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:46.136344 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 130850us
W20260620 19:58:46.138320 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 46427.3us
I20260620 19:58:46.139642 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:46.117642 7954 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.143898 7941 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.145829 6378 raft_consensus.cc:1240] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:46.145756 7884 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.147718 7948 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.148877 6911 raft_consensus.cc:1240] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.142556 6198 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.117695 7873 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.149830 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.132550 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.152357 6877 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:46.153153 7733 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:46.153121 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.153685 7733 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:46.153837 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:46.154070 7733 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:46.154538 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:46.154853 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.157796 6271 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.142277 7957 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:46.148322 7927 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.152344 7873 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.167840 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:46.156158 7928 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.164960 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.152168 7960 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.155052 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.168795 6198 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.161787 7933 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.161665 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.177385 7911 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.190646 7963 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.194659 7884 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.191870 7965 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.196799 7927 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:46.193115 7962 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.183267 7937 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.200130 7960 consensus_queue.cc:1059] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260620 19:58:46.203941 7963 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.185096 7933 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.219199 7733 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.201259 6904 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:46.225548 7733 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.212944 7933 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.231033 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.231292 6223 raft_consensus.cc:1240] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.231690 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.223039 6906 raft_consensus.cc:2393] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:46.232177 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.232860 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.212970 7957 consensus_queue.cc:1059] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260620 19:58:46.244338 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.206542 7917 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.264016 7941 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.257705 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.212121 7925 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.276166 7925 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.278666 7948 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.279891 6376 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.281188 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.282323 7961 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.284138 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.284142 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.287024 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.287457 7873 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.288394 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.288800 6904 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.291605 6372 raft_consensus.cc:1240] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.294199 6302 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:46.294716 7968 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.296718 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.297653 6907 raft_consensus.cc:2393] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:46.299010 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.300664 7921 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.302394 6232 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:46.302762 6378 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: []
W20260620 19:58:46.303926 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 144220us
I20260620 19:58:46.306103 6271 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:46.303411 7961 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.307260 7851 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.309041 7852 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.309443 7852 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:46.309825 7852 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.310837 6299 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.312402 7917 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:46.314163 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.309468 7851 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.315011 7935 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.316365 6372 raft_consensus.cc:1240] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:46.316865 7920 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.317952 6374 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.310130 7931 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.320037 6908 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.320914 7931 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.316999 6271 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:46.323983 6271 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.324443 7851 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.324923 7851 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:46.325371 7851 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.326156 7968 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.326527 7968 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.326881 7968 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.326917 7921 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.325155 7889 consensus_queue.cc:1059] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:46.328259 6271 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.330796 6271 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.269168 7962 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.307586 7961 consensus_queue.cc:1059] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260620 19:58:46.331949 7963 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.352347 6909 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:46.358763 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.361420 6302 raft_consensus.cc:1240] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.362733 6908 raft_consensus.cc:1240] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.320290 7852 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.338734 7964 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.330130 7889 consensus_queue.cc:1059] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260620 19:58:46.367940 6374 raft_consensus.cc:1240] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.328243 7914 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.368072 6300 raft_consensus.cc:1240] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.369249 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.370857 6271 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.371969 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.338860 7965 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.375919 6378 raft_consensus.cc:2393] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 2: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:46.376212 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.358345 7963 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.369971 7884 consensus_queue.cc:1059] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260620 19:58:46.378571 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.373997 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.380267 7852 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.380592 6372 raft_consensus.cc:2393] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:58:46.381214 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 133485us
I20260620 19:58:46.379079 7889 raft_consensus.cc:3055] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.379254 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:46.387943 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 298650us
I20260620 19:58:46.390285 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.370399 7964 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.366511 7921 consensus_queue.cc:1059] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260620 19:58:46.389436 6875 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.379113 7969 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:46.379333 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.392557 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.377266 7962 raft_consensus.cc:3055] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.385447 7889 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.389662 6195 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.396605 6875 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.383782 7963 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.381669 7920 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.370659 7915 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.402375 7973 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.402437 7873 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.402951 7935 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:46.403082 7918 raft_consensus.cc:3055] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.391652 7957 raft_consensus.cc:3055] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.385047 7975 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.406098 6195 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:46.406131 6875 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.396103 7964 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.397642 7917 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.395681 7911 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:46.418046 7913 raft_consensus.cc:3055] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.418423 7913 raft_consensus.cc:740] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.404160 7948 consensus_queue.cc:1059] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260620 19:58:46.419296 7913 consensus_queue.cc:260] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.411525 7975 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.420532 7913 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.420928 7975 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.408936 6271 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.407922 7961 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.398459 7962 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.409749 6378 consensus_queue.cc:237] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:46.409039 7918 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.414788 6379 raft_consensus.cc:1240] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.415854 7954 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:46.416174 7914 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:46.410620 7957 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.399032 7940 consensus_queue.cc:1059] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260620 19:58:46.417755 7911 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.387638 7978 consensus_queue.cc:1059] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
W20260620 19:58:46.407233 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:46.423790 6299 raft_consensus.cc:1240] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.421981 7946 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.428514 7937 raft_consensus.cc:3055] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.432881 7937 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.433744 7937 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.434785 7978 raft_consensus.cc:3055] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:46.435395 7980 raft_consensus.cc:3055] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.435185 7978 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.435768 7980 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.436295 7911 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.436513 7978 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.436643 7980 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.437889 7978 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:46.438010 7980 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.425808 6195 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:46.446856 6195 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.447971 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.449062 6195 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:46.450773 7959 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.450898 7961 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:58:46.451035 7946 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:46.453290 7946 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.453666 7946 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:46.455349 7915 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.428387 7912 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:46.456384 7912 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.456857 7912 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:46.451156 7977 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.450572 6195 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.460508 6195 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.463006 7961 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.431092 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.463416 7961 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.463826 7961 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.420549 7933 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.431682 7940 raft_consensus.cc:3055] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.468429 7940 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:58:46.468914 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:46.469337 7940 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.425348 7917 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:58:46.470954 7917 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.471419 7917 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:58:46.424564 6228 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:46.440623 7937 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.440779 7962 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.428799 7979 consensus_queue.cc:1059] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260620 19:58:46.427551 7889 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.485327 7962 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.485625 7889 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.485441 7946 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.487504 7946 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.440788 7957 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.488415 7957 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.444434 7918 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.427960 7954 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.466768 7977 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:58:46.476202 6195 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.494957 6195 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:46.428053 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:46.475219 7940 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.477522 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:58:46.498914 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 70909.2us
I20260620 19:58:46.499773 6271 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.482167 7968 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:46.480480 7963 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.491734 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.496620 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.496668 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.512212 7982 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:46.512730 7982 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.513172 7982 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:46.466550 7933 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.501695 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.503723 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.517182 7918 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.500484 6875 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.518450 7873 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.518842 7873 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.506494 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:46.519202 7873 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.520538 7977 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.521188 7977 consensus_queue.cc:1059] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:58:46.499333 7976 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.494159 7935 consensus_queue.cc:1059] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260620 19:58:46.522358 7959 raft_consensus.cc:3055] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:46.523463 7959 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.523694 7973 raft_consensus.cc:3055] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.524067 7973 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.524343 7959 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.524889 7973 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.525586 7959 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:46.526140 7973 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.515550 6874 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.522915 6195 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.527565 7979 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.527995 7979 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.528468 7979 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.528519 7931 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:46.529285 7733 consensus_queue.cc:1059] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:46.522766 7976 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.509313 7948 consensus_queue.cc:1059] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:46.528028 7960 raft_consensus.cc:3055] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.519222 6875 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.489737 7917 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.517557 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:46.529112 6874 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.497144 7954 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.534897 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.540373 6271 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.544214 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.494498 7851 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.547086 7948 raft_consensus.cc:3055] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.547276 7911 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.547511 7948 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.547706 6271 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.548118 7935 raft_consensus.cc:3055] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.548561 7733 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:58:46.548789 6271 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:46.511438 7946 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.549422 6875 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:58:46.540925 7929 consensus_meta.cc:287] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.646s user 0.013s sys 0.015s
I20260620 19:58:46.529084 7931 raft_consensus.cc:697] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.533525 7933 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.554231 6875 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.505756 7981 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.533378 7960 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:58:46.559298 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:46.531847 7976 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.548616 7965 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:46.551347 7935 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.563128 6875 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.563618 7852 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:46.564045 7733 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.564157 7852 raft_consensus.cc:697] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:46.564393 7934 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:46.564644 7733 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.564903 7934 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.565173 7733 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.565423 7928 consensus_queue.cc:1059] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:46.565266 7852 consensus_queue.cc:237] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.552551 7911 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:46.556277 7927 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.567354 6875 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.573608 7966 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.580122 7733 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.558772 7981 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:46.572614 7927 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.583233 7927 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.582969 7931 consensus_queue.cc:237] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.584162 7968 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:46.565266 6903 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: []
I20260620 19:58:46.586247 6875 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.562170 6303 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.562397 7965 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:46.565285 7934 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:46.566216 7963 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:46.590270 6198 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:46.590667 7733 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.557829 7929 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.567802 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:46.595319 7733 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.551453 7948 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.590106 7954 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:46.585496 7981 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.601145 7733 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.591810 7965 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.603354 6907 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:46.564041 7935 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.593376 7960 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.599350 7851 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.605803 7948 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.641471 7972 raft_consensus.cc:3055] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.643314 7972 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.624400 7961 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
W20260620 19:58:46.647112 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:46.637770 7917 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.642722 7929 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:46.659242 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 102578us
I20260620 19:58:46.665098 6345 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.672078 7960 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.675745 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.676961 7935 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.676765 7914 consensus_queue.cc:1059] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:46.678311 7914 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.679019 7914 raft_consensus.cc:3055] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.679334 7914 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.680218 7914 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.666625 7925 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.685096 6372 raft_consensus.cc:1240] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.645375 7972 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.686996 6373 raft_consensus.cc:1240] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:58:46.687597 7972 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:58:46.670318 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 121575us
I20260620 19:58:46.683437 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.695672 6299 raft_consensus.cc:1240] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.701195 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.695696 7921 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:58:46.695878 7977 consensus_queue.cc:1059] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260620 19:58:46.703488 7977 raft_consensus.cc:3055] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.703811 7977 raft_consensus.cc:740] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.704008 7925 consensus_queue.cc:1059] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:58:46.706063 6378 raft_consensus.cc:1240] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.686095 7975 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.704979 7921 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:58:46.704643 7977 consensus_queue.cc:260] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.706861 7921 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.707432 7977 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.707742 7921 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.711382 6298 raft_consensus.cc:1240] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.711884 7921 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:58:46.712486 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.712291 7927 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.713973 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.714890 6303 raft_consensus.cc:1240] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.717085 7982 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.718097 6299 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:46.704463 7979 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.719112 7982 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.718820 7964 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.728010 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.720232 7979 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.733198 6374 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:46.709107 7914 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.733881 6374 raft_consensus.cc:606] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:46.739830 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.740546 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.740685 7927 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.740746 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.741720 6298 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:58:46.742137 7983 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260620 19:58:46.744163 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.747488 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.748204 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.749812 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.705809 7978 consensus_queue.cc:1059] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260620 19:58:46.750844 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:46.747524 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.742584 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:58:46.754927 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 168694us
I20260620 19:58:46.736531 7984 consensus_queue.cc:1059] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260620 19:58:46.765795 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.766644 6302 raft_consensus.cc:1240] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.755726 6231 raft_consensus.cc:1240] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.756186 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.756732 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:46.757285 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.757776 7912 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.759259 7961 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:46.773592 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.774351 6301 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:46.774626 7873 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.761104 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.761629 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.763640 6875 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.763988 6232 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:46.764707 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.764667 6299 raft_consensus.cc:1240] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:46.770001 6229 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:46.748296 7964 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.778370 6875 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.778518 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:46.776989 6345 consensus_peers.cc:597] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42a982b7017a41e1b0ca419a82cacf59. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:46.779146 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:58:46.783483 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 58715.5us
I20260620 19:58:46.783828 6307 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:46.780059 6907 raft_consensus.cc:1240] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.783891 6875 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.780788 7984 raft_consensus.cc:3055] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.785043 6306 raft_consensus.cc:2393] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:46.785382 6875 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:46.778424 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.785084 7985 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.786252 7980 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:46.786978 6195 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.787833 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.789929 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.790251 6195 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.746500 7976 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.791344 7978 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.791265 7980 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:46.788615 7957 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:46.814924 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.793568 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.816931 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.817080 7965 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.817955 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.818619 6225 raft_consensus.cc:2393] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 2: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:46.819020 7965 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:46.819142 7975 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.819305 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.793931 7937 consensus_queue.cc:1059] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:46.794127 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.820740 6230 raft_consensus.cc:2393] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 2: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:46.820752 6875 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.794961 7946 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.776717 7873 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.826035 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.827123 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.827092 6299 raft_consensus.cc:1240] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.797338 6345 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:46.798678 7991 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:58:46.786471 7929 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:46.798993 6908 raft_consensus.cc:1240] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:46.835120 7991 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.800665 6195 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.785264 7984 raft_consensus.cc:740] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.801728 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.763173 7927 consensus_queue.cc:1059] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260620 19:58:46.802774 7962 consensus_queue.cc:1059] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:46.840868 7915 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:46.841044 7986 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.805975 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.808833 6226 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:46.809639 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.842689 7927 raft_consensus.cc:3055] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.843065 7927 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.841809 7925 consensus_queue.cc:1059] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260620 19:58:46.843839 7927 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.844401 7915 raft_consensus.cc:697] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.844519 7920 consensus_queue.cc:1059] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:46.845194 7927 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:46.845824 7838 raft_consensus.cc:3055] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.845376 7915 consensus_queue.cc:237] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.811020 6379 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:46.841372 7962 raft_consensus.cc:3055] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:46.846969 7962 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.848131 6907 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.848311 7925 raft_consensus.cc:3055] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:46.848824 7925 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.811160 7918 consensus_queue.cc:1059] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:46.812238 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.813299 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.813390 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.791697 7961 consensus_queue.cc:1059] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260620 19:58:46.814306 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.852867 6306 raft_consensus.cc:1240] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:46.815686 6304 raft_consensus.cc:1240] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.793041 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:58:46.815804 7980 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.820889 6224 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:46.795061 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.822475 6228 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:46.823850 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.825805 7889 raft_consensus.cc:3055] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:46.831719 7972 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:58:46.831650 6875 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.832990 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.832878 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.859958 6301 raft_consensus.cc:2393] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:46.834578 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:46.799846 7921 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.849861 7925 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.839018 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.846171 7838 raft_consensus.cc:740] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.835443 7929 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:46.813869 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.851686 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.854914 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:46.883065 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:46.851904 6345 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:58:46.855715 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 61941.8us
I20260620 19:58:46.851047 6303 raft_consensus.cc:2393] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:46.860713 6875 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.831205 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.850579 7918 raft_consensus.cc:3055] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.861800 7948 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.841646 7991 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.867172 6298 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.857080 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.894415 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.868041 7925 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:46.868907 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.869395 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.869843 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.870016 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:46.876618 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.884794 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.901769 6232 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:46.885414 6299 raft_consensus.cc:2393] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:46.899547 6307 raft_consensus.cc:1240] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:46.900753 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.886013 6875 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.857391 7889 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:46.894290 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.894668 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.867317 7838 consensus_queue.cc:260] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.904382 6875 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.907720 6271 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.903767 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.904196 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.920020 7935 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.923251 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:46.900703 7921 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:46.925920 6904 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.926348 6344 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:46.888260 7918 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.888259 7972 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.923223 7838 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:46.923408 7914 consensus_queue.cc:1059] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260620 19:58:46.941094 7914 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.923202 7980 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.942129 6908 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:46.943600 7889 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.944976 7889 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:46.945223 6377 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:46.945930 7914 raft_consensus.cc:3055] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.922314 7981 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.932533 7941 consensus_queue.cc:1059] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260620 19:58:46.936925 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.946245 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.937875 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.939248 7912 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.952446 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.946286 7914 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.953502 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.953732 7914 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.894251 7929 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.954494 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.955317 7965 consensus_queue.cc:1059] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:46.956386 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.956687 7965 raft_consensus.cc:3055] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:46.957064 7965 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:46.957352 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:46.957815 7965 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.939416 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:46.959959 7976 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:46.939618 6380 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:46.947183 7941 raft_consensus.cc:3055] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:46.899060 7984 consensus_queue.cc:260] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:46.895551 7934 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:46.951067 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:46.951332 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:46.973747 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 46043.5us
W20260620 19:58:46.974089 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 66427.3us
I20260620 19:58:46.996651 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.997728 6271 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:58:46.987066 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:46.923509 7992 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:46.918363 7948 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:46.990639 6910 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:47.003196 6271 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:46.972508 6345 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:46.992107 6877 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:58:46.995229 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 68946.9us
I20260620 19:58:47.004397 6344 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.004710 6345 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:47.005411 6344 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:47.001405 6875 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.008054 6875 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.010311 6875 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:46.989105 7941 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:46.990362 7912 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.015777 6196 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:47.013088 7921 raft_consensus.cc:697] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:47.016196 7990 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:47.011876 7873 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:47.000555 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:47.000706 7852 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.006103 7964 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.011739 7972 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:47.032969 6298 raft_consensus.cc:2393] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:47.017290 7914 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.024977 7980 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:47.025689 7960 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.046561 7972 raft_consensus.cc:697] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:47.046653 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:47.047624 7972 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.026011 7941 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:47.028285 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:47.010564 7965 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.016983 7918 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:47.009023 7991 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:47.028779 7984 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.022257 6271 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:47.011458 7931 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.037959 6227 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:47.017094 7917 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:47.017709 7913 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.023347 7838 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:47.071017 7838 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:47.038959 6223 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:47.045132 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:47.051086 6198 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:47.028193 7934 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:47.062099 6911 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:47.063077 6298 raft_consensus.cc:1240] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:47.076637 6875 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:47.013741 7962 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.051023 7990 raft_consensus.cc:697] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:47.062810 6128 catalog_manager.cc:5696] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:47.079336 6372 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:58:47.064016 7918 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.064929 7941 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.072450 7933 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
W20260620 19:58:47.067859 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 45644.4us
I20260620 19:58:47.069249 6910 raft_consensus.cc:1240] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:47.052325 7981 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:47.010301 7995 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:47.085295 6271 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:47.082389 7966 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:47.077708 7960 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.087571 7987 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:47.090169 7994 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.091380 6271 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:47.079464 7990 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.087812 7933 raft_consensus.cc:697] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:47.092042 7917 consensus_queue.cc:1059] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:47.093688 7987 raft_consensus.cc:697] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:47.087662 7962 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:47.072180 7838 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.096474 7912 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:47.102473 7912 raft_consensus.cc:697] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:47.102523 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:47.103549 7912 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.102494 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:47.095759 7960 consensus_queue.cc:1059] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260620 19:58:47.094852 7966 raft_consensus.cc:697] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:47.094715 7987 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.101651 7917 raft_consensus.cc:3055] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:47.115767 7917 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:47.101873 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:47.100988 6271 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.091097 7995 raft_consensus.cc:697] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:47.106267 7960 raft_consensus.cc:3055] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:47.122136 7960 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:47.123018 7960 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.124714 7960 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.118929 7917 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.119145 7966 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.127381 7917 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.093716 7933 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.119102 7995 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:47.135864 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:47.181281 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 61910.9us
W20260620 19:58:47.182421 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 81521.1us
I20260620 19:58:47.184224 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.186013 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:47.185773 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 181091us
I20260620 19:58:47.083739 7921 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.185006 7970 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.192636 6911 raft_consensus.cc:1240] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:47.193215 6877 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.191372 6271 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.182024 7981 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.202354 6227 raft_consensus.cc:1240] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
W20260620 19:58:47.189951 7997 log.cc:927] Time spent T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260620 19:58:47.203099 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.203140 6877 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.232424 7927 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:58:47.233858 6271 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.234930 6271 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.240432 7911 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.240514 7927 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:47.241823 7968 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.243016 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:47.243530 6372 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.243742 7991 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:47.192263 7851 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:47.245041 6271 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.244889 6877 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.246389 7991 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.243027 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:47.248826 7911 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.248842 7973 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.259413 6877 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.260979 7973 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:47.241784 7927 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.262683 6271 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.263998 6271 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.264819 7852 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:47.265424 6271 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.248335 7972 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260620 19:58:47.270745 7968 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:58:47.271472 6298 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:47.271587 7991 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:47.275362 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:47.275622 6911 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.276122 6904 raft_consensus.cc:2393] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:58:47.084519 7976 consensus_queue.cc:1175] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" responder_term: 4 status { last_received { term: 3 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260620 19:58:47.278244 7981 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.240093 7990 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.232859 7980 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:47.279740 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.280869 7980 raft_consensus.cc:697] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:47.281466 7973 consensus_queue.cc:1059] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:47.281741 7980 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.282446 7973 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:47.282756 7990 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:47.285638 6230 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:47.281787 7968 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.290000 6223 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:47.286767 7973 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:47.292523 7968 consensus_queue.cc:1059] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:47.292848 7973 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.293555 7965 raft_consensus.cc:3055] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:47.293924 7965 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:47.294212 7973 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.265873 7975 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.294719 7965 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:47.295588 6298 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.279764 7918 consensus_queue.cc:1059] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260620 19:58:47.298779 7934 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:47.298789 7918 raft_consensus.cc:3055] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:47.300745 7918 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:47.301548 7918 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.307104 6223 raft_consensus.cc:1240] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:47.308001 7994 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.309463 6228 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.312320 7981 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:47.316830 6271 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:47.318468 7990 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:47.319018 7990 raft_consensus.cc:697] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:47.319916 7990 consensus_queue.cc:237] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:47.336741 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 331306us
I20260620 19:58:47.337723 6344 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:58:47.345325 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:47.360453 6344 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:47.367911 6910 raft_consensus.cc:1240] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:47.347046 8001 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.376665 6344 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.378925 7914 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.382355 8001 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:58:47.383224 7965 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.383587 7914 consensus_queue.cc:1059] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:47.384654 7925 raft_consensus.cc:3055] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:47.385000 7925 raft_consensus.cc:740] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:47.385804 7925 consensus_queue.cc:260] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:47.386770 7998 consensus_queue.cc:1059] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:47.388538 6344 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:47.389400 7915 raft_consensus.cc:3055] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:47.389850 7915 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:47.390197 8001 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.390714 7915 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.393432 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:47.393977 7941 consensus_queue.cc:1059] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:47.392496 6344 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:47.396178 6195 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:47.394879 7994 consensus_queue.cc:1059] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260620 19:58:47.404623 6344 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.405800 7976 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.407349 7913 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:58:47.410859 7994 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.413496 7918 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:47.448118 6227 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.450475 6911 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.464798 7920 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260620 19:58:47.466082 6372 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.468544 7915 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.472263 6227 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.474350 7917 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:47.477595 6299 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.479485 7966 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:47.485672 7966 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:47.486742 7925 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.485642 6227 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.503114 7838 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260620 19:58:47.504501 7976 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260620 19:58:47.575450 8009 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260620 19:58:47.575471 6910 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.579174 7933 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:47.601842 6232 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260620 19:58:47.628268 7915 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:58:47.631583 8001 raft_consensus.cc:3055] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:47.631979 6907 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.632042 8001 raft_consensus.cc:740] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:47.632997 8001 consensus_queue.cc:260] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:47.634163 8001 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.634019 7933 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:47.635787 6229 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.639551 7966 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:47.644168 6380 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.658627 6229 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260620 19:58:47.661625 8014 log.cc:927] Time spent T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260620 19:58:47.649868 7927 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260620 19:58:47.663861 7966 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:58:47.673439 6229 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.678944 6910 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.686322 6910 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.688055 7925 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:47.689954 7927 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:47.690573 7927 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.723064 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:47.732628 7976 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:47.742872 6910 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.755934 6375 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:47.750797 7991 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:47.764317 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:47.764923 6378 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:47.764777 7991 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.766824 7991 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:47.767819 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:47.761792 7933 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260620 19:58:47.768774 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:47.769443 6378 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:47.770612 6198 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:47.771661 6877 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:47.774704 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:47.775285 6910 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:47.778108 6229 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.779960 7965 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:47.792876 7980 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:47.764048 7927 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:47.764273 6373 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.776461 7966 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:47.755558 7976 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.776693 7991 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:47.802246 7966 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:47.835515 7991 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:47.849464 7991 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:58:47.851017 8006 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260620 19:58:47.853209 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:47.854115 6910 raft_consensus.cc:2393] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:47.862150 7976 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:47.796499 7980 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.868564 6375 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.852713 7966 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:47.870051 7921 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:47.870534 7921 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.871860 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:47.872081 6229 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:47.872713 6375 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:47.874452 6198 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:47.875797 7976 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:47.876250 7976 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:47.876672 7976 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:47.878252 6228 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260620 19:58:47.879966 8010 log.cc:927] Time spent T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
I20260620 19:58:47.872081 7980 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:47.914065 7934 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:47.914758 7934 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.917240 7934 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:47.917824 7934 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:47.943975 6344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.004s sys 0.021s
I20260620 19:58:47.918340 7934 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:47.904091 7925 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
W20260620 19:58:47.955065 6344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s user 0.004s sys 0.022s
W20260620 19:58:47.965406 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 560766us
I20260620 19:58:47.967146 6344 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:47.972077 6344 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.973477 6344 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:47.975294 8018 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:47.954120 6229 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:47.977111 6344 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.974079 7915 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:48.003408 6344 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:48.008093 6344 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:47.976274 7941 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:47.974303 8001 consensus_queue.cc:1059] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260620 19:58:47.981247 7970 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:48.015105 7995 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:48.009260 8018 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:48.009522 7838 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:48.018278 6344 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:58:48.025823 8022 log.cc:927] Time spent T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20260620 19:58:48.027557 8021 log.cc:927] Time spent T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260620 19:58:48.026520 6344 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:48.031973 7995 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:48.033094 6344 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:48.033167 7991 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:48.035377 8020 log.cc:927] Time spent T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260620 19:58:48.036854 7915 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:48.043884 7921 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.046835 7995 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:48.019636 7970 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.051357 7995 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:58:48.052280 7995 raft_consensus.cc:3055] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:48.052311 7970 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.041265 7934 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.052733 7995 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:48.073907 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.075985 6908 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:48.046504 7966 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.082194 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.085198 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.083499 7990 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:48.084012 7976 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.085435 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:48.104522 8023 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.079785 7925 raft_consensus.cc:3055] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:48.108537 6375 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260620 19:58:48.110002 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.111534 6378 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:48.113276 6271 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:48.085218 7921 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.110741 7925 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:48.107025 6906 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:48.106124 7991 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.160931 6908 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:48.107157 6372 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:48.113315 7980 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260620 19:58:48.104938 7990 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.107676 8023 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.160290 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.154389 7995 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.153908 7921 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.200858 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:48.203828 6909 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 6
I20260620 19:58:48.197932 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.204991 6377 raft_consensus.cc:2393] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:48.207695 6375 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:48.207860 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:48.212304 6377 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:48.175608 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.194765 7991 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.205693 7927 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260620 19:58:48.220603 6909 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:58:48.199221 8038 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:48.234213 6196 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:48.163395 7925 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:48.259428 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 93642.7us
W20260620 19:58:48.260962 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 62669.3us
W20260620 19:58:48.229017 8026 log.cc:927] Time spent T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20260620 19:58:48.230544 6373 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:48.241926 8038 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:48.304814 6228 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92212144534e4394844003016ba67a47"
dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
mode: GRACEFUL
new_leader_uuid: "39662d01be7a44f682d160227f457661"
from {username='slave'} at 127.0.0.1:50828
I20260620 19:58:48.313549 6228 raft_consensus.cc:606] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Received request to transfer leadership to TS 39662d01be7a44f682d160227f457661
I20260620 19:58:48.313901 6228 raft_consensus.cc:614] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:58:48.267544 6375 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260620 19:58:48.294492 8029 log.cc:927] Time spent T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.236s user 0.000s sys 0.001s
I20260620 19:58:48.361488 8045 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.245385 7991 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.301994 7925 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:58:48.365983 8027 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260620 19:58:48.367988 7976 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:48.370138 8034 log.cc:927] Time spent T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
W20260620 19:58:48.374547 8030 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260620 19:58:48.363665 7941 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260620 19:58:48.308743 7975 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.376016 7921 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.302917 8042 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:48.361380 8056 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.374760 7915 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.304059 8001 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260620 19:58:48.379310 6377 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
W20260620 19:58:48.359673 8028 log.cc:927] Time spent T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.280s user 0.001s sys 0.001s
W20260620 19:58:48.384429 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 150295us
I20260620 19:58:48.328801 8038 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:58:48.387202 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 273948us
I20260620 19:58:48.396939 6224 raft_consensus.cc:1275] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:48.401589 7975 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.421195 7975 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.423900 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.424736 6299 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:48.427878 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.428670 6307 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:48.400861 6228 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:48.376689 7976 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.380422 8045 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.433538 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:48.408027 7965 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:48.408599 7990 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.437057 6301 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 4
W20260620 19:58:48.439885 8059 log.cc:927] Time spent T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260620 19:58:48.414022 8047 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:48.452471 8057 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:48.428134 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:48.435174 8055 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:58:48.435591 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.472199 7838 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:48.436793 7976 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.411698 7915 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:48.476795 6877 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:48.403211 8042 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.447170 7965 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.500725 6301 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
W20260620 19:58:48.505998 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 89195.4us
I20260620 19:58:48.428825 8040 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260620 19:58:48.446494 8023 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:48.510857 6225 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:58:48.513573 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 110300us
I20260620 19:58:48.405963 8056 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.504827 8055 raft_consensus.cc:697] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:48.506392 7917 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.520838 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:48.526961 6907 raft_consensus.cc:2393] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:48.530616 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.540249 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.515091 8061 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:48.540994 6301 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:48.509573 7933 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260620 19:58:48.532061 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:48.547158 6230 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:48.549705 6909 raft_consensus.cc:2368] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4.
I20260620 19:58:48.462540 8047 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.515517 7990 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260620 19:58:48.478160 8057 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.522846 7917 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.545411 8061 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:48.555389 7991 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.559499 8052 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.527566 8060 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260620 19:58:48.584188 7965 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.560536 8044 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:48.585866 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:48.569231 8061 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:48.589737 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:48.569692 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.592042 6224 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:48.588438 6232 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:48.605602 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.606613 6377 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:48.614087 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.614842 6377 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:48.578621 8040 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260620 19:58:48.616195 6198 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:48.590334 7991 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.631431 7966 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.633881 7941 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.634408 7941 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.635078 8067 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.635190 8056 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.635761 8056 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.635622 8067 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:48.636261 8056 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.625275 7991 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.638767 7925 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.608413 6380 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:48.618654 8045 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.636436 7941 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:48.648248 6377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a56f959cee394e398810b9170b9d24be"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:58:48.575066 8055 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:48.651554 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 123797us
I20260620 19:58:48.653528 6377 raft_consensus.cc:606] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:48.653962 6377 raft_consensus.cc:614] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:58:48.626080 6379 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:48.655776 6225 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260620 19:58:48.587374 8052 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.653652 6877 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:48.593876 8044 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:48.645313 8043 log.cc:927] Time spent T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260620 19:58:48.659361 8056 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.643278 8042 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.647670 7976 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.639222 7925 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.651373 8045 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:48.647883 7915 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:48.654760 7966 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.660287 6877 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:48.661405 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.685848 6196 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:48.667876 8047 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.719187 7927 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.664817 7995 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:48.667768 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.660670 8053 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:48.662016 7917 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.661144 8057 leader_election.cc:290] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.731158 8056 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.679623 8045 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:48.669166 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.719753 7927 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:48.742818 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 57050.9us
I20260620 19:58:48.723274 8064 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.726655 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.682879 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.728816 8053 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:48.661144 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.691901 8060 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.698465 7980 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:48.736042 6910 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260620 19:58:48.684729 7966 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.741262 8047 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:48.683611 8038 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.743223 8045 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:48.743348 8056 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.759333 6905 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:48.755352 8064 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.770692 6906 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:48.771649 8064 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.772872 6196 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:48.773690 8069 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:48.766757 8067 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:48.753834 8053 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:48.745049 8057 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:58:48.774268 8069 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:48.752764 8070 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.760051 7980 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:48.768254 7966 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:48.778653 8047 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.785449 6298 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:48.785590 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.797883 6902 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:48.785893 8070 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.778003 7927 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.784310 8058 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.769246 7925 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.769381 8052 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.796229 8047 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:48.799145 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 382305us
I20260620 19:58:48.781142 8069 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:48.798918 6196 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:48.825166 8041 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.825819 8041 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.828375 8041 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.830327 8065 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:48.831597 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.832428 6377 raft_consensus.cc:2393] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:48.832602 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.833403 6373 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:48.783529 7991 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:48.784881 7980 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:48.837621 8047 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260620 19:58:48.836545 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 292403us
I20260620 19:58:48.841440 6377 raft_consensus.cc:1275] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260620 19:58:48.839555 6874 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:48.768371 8061 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.849094 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.849674 6377 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:48.856962 6271 leader_election.cc:400] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4.
I20260620 19:58:48.857349 6271 leader_election.cc:403] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:58:48.845872 8047 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:48.781323 8060 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.823458 8045 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.847560 7927 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:48.848712 8057 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.866914 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.872210 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.872411 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.874388 6301 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:48.874572 6305 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:58:48.872951 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 255661us
I20260620 19:58:48.855034 8065 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.878489 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.879050 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.862053 7925 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.858860 7995 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.852626 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.880879 6377 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:58:48.855432 7987 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.878386 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.865061 8047 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:48.931317 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:48.932345 6902 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:48.922569 8065 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.933797 6372 raft_consensus.cc:2410] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 4 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260620 19:58:48.863328 7970 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.936225 7970 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:48.836841 8058 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:48.873555 6300 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:48.918558 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.865224 7966 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.921945 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.993108 6198 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:48.929409 8037 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260620 19:58:48.919744 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:48.865351 8048 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4.
I20260620 19:58:48.926154 8042 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:48.930724 7987 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.929013 7925 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:48.922106 7838 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.922950 7976 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.927029 8064 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:48.943674 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:48.943403 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:48.882313 7927 raft_consensus.cc:697] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:49.029999 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:48.883162 8044 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.010958 6305 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:48.939857 6196 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.022957 6198 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:58:49.131400 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 108542us
I20260620 19:58:49.132366 6198 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:48.922501 8067 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:48.880864 8057 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.025884 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.027776 6224 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
W20260620 19:58:49.030567 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 159188us
I20260620 19:58:49.030732 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.030963 7925 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:49.133913 6300 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:49.111956 6273 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:49.134783 8041 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:49.134846 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:49.135190 8041 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.135460 6300 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.135545 8041 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:48.939139 7995 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.137521 8037 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:49.009917 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.138195 8037 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.139020 6301 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:49.036530 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:48.997604 8052 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.020493 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.040189 8060 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:49.042555 8056 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.159334 8056 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:58:49.159788 8056 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.160202 8056 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:58:49.043685 8058 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.044791 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.020946 6298 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:49.053217 8080 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.055835 8050 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:49.059372 8038 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.062431 8061 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:48.995756 7966 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.046229 6379 raft_consensus.cc:2393] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:49.181064 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.181735 6906 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:49.045746 7838 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.026315 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.066839 8045 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.067752 8070 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:49.068608 8074 log.cc:927] Time spent T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260620 19:58:49.073210 8078 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.045392 8065 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.077294 7970 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:49.023910 8042 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.022351 7991 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:49.047719 7976 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.083606 8069 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.084899 8047 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.089377 7980 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.094010 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.094830 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.108744 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:49.111835 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.193537 6223 raft_consensus.cc:2410] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260620 19:58:49.118659 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 479449us
I20260620 19:58:49.119385 6225 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
W20260620 19:58:49.130355 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 190532us
I20260620 19:58:49.138764 7964 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:49.038735 7987 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.139925 6307 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:49.140945 8071 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:49.142168 8067 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.143893 7925 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.144289 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.245733 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.245918 6301 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:49.135322 6273 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:49.146790 8037 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.151757 6300 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:49.152346 8053 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.153304 8055 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.155166 8057 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.158010 6344 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:49.158584 8039 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.158819 7927 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.168857 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.168663 8080 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.169584 8050 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.179324 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.261511 6910 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:49.179610 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:49.180698 8088 log.cc:927] Time spent T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.071s user 0.002s sys 0.000s
I20260620 19:58:49.181831 7966 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:49.052587 8064 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.267163 8067 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.267421 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:49.267676 8067 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:49.268322 7933 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.268239 8067 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.268452 8064 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.268815 7933 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.269225 7933 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.270308 8067 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.163478 6229 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:49.184016 7995 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.274042 8044 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:49.274538 8044 raft_consensus.cc:697] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:49.114990 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.163666 8052 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.278635 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.188827 6380 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:49.207430 6376 raft_consensus.cc:2410] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 4 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260620 19:58:49.256431 8038 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.210821 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.283758 6299 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:49.211440 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.284520 6911 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:49.284773 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.283793 6306 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:49.244238 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.207998 8060 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:58:49.250365 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 115119us
I20260620 19:58:49.260398 8057 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:58:49.254134 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 96225.1us
I20260620 19:58:49.263818 8091 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:49.262888 6909 raft_consensus.cc:2368] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4.
I20260620 19:58:49.271765 8053 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.249876 7925 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.141050 6372 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:49.203341 6195 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:49.206148 6377 raft_consensus.cc:2393] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:49.279134 6374 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:49.253676 8055 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:49.190045 8070 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:49.317907 8092 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:49.317737 8070 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.193245 8078 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.318817 8078 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.317775 8055 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:49.323804 7975 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.324317 7975 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.326315 7975 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.328985 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:49.330315 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.247841 8071 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.331279 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.330953 8044 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.330988 6304 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:49.332531 8071 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:49.333309 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.333947 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:49.334520 6304 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.335620 8070 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.333336 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.250838 8063 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:49.336786 6300 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:49.337133 8063 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.337646 8052 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.288776 6196 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.199226 7991 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.339011 8047 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.339233 8063 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.340061 7991 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.340744 8050 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:49.340909 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 52209.2us
I20260620 19:58:49.341795 6304 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:49.342242 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.342931 6372 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:49.343118 7991 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:49.343508 7991 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.343842 7991 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:58:49.346081 8097 log.cc:927] Time spent T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260620 19:58:49.294046 7995 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.347092 7995 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.348381 6273 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:49.283612 8038 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.302392 6874 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:49.383545 8038 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:58:49.383791 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 81432.8us
I20260620 19:58:49.384503 8041 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.385233 6874 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:49.385340 8055 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.316534 8069 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:49.315797 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 112538us
I20260620 19:58:49.295490 8091 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.387018 7966 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.387429 7966 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.387764 7966 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.389145 6195 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.275007 8039 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.336244 7976 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.335662 7925 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.292615 6344 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:58:49.403288 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 110759us
I20260620 19:58:49.289728 8060 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.294727 6910 raft_consensus.cc:2393] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:49.351377 6271 leader_election.cc:400] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4.
I20260620 19:58:49.351399 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.405385 6271 leader_election.cc:403] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260620 19:58:49.286890 6302 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
W20260620 19:58:49.366900 8100 log.cc:927] Time spent T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260620 19:58:49.371208 8044 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:58:49.374794 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 242491us
I20260620 19:58:49.411805 6198 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:49.349480 7995 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:49.313819 7838 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.378297 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.386466 8091 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:49.379061 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:49.412087 8063 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.418845 6903 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:49.419082 8063 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.419584 8063 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.420019 7975 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.420503 7975 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:49.379416 8090 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260620 19:58:49.408537 8052 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:49.416962 8047 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:49.380155 8042 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.380707 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.424582 6905 raft_consensus.cc:2393] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:49.424654 8060 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.381891 8056 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.383078 7980 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.392768 6305 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:49.442559 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:49.393177 6306 raft_consensus.cc:2393] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:49.394336 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.442994 8056 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.422386 7975 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.447458 8052 raft_consensus.cc:697] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:49.447837 8091 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.448784 8052 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.449846 8091 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:49.406075 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 54888.6us
W20260620 19:58:49.407660 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 198624us
I20260620 19:58:49.409953 8099 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.414034 6909 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:49.378391 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:49.453145 6345 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.453577 6906 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.415529 6198 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:49.416921 8044 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.417472 8041 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.410713 7964 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.417868 8080 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.293974 8057 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.402413 8069 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.428956 6910 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:58:49.435345 8055 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.436626 7933 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.418380 8039 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.455081 6198 leader_election.cc:304] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:49.458138 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.459519 6344 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:49.459895 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.460707 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.458312 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.402508 8040 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.468052 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:49.556356 6232 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
W20260620 19:58:49.556504 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 97047.2us
I20260620 19:58:49.555615 6302 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:49.557159 6344 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:58:49.558243 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 103229us
I20260620 19:58:49.463165 6345 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:49.463678 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.464419 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.440627 7966 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.559268 6228 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:58:49.559949 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.564764 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.566010 6910 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:49.566262 6231 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 1.
I20260620 19:58:49.563225 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:49.557679 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.567081 6232 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:49.516742 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.561851 6911 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
W20260620 19:58:49.540153 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 71300.1us
I20260620 19:58:49.516161 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.568539 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.565387 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.561944 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:49.559059 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 206633us
I20260620 19:58:49.434392 8078 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.455360 8099 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.770512 8057 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.772779 7933 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.766041 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.784428 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:49.667601 8055 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:49.785825 8055 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.778234 7966 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:49.787452 7964 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4.
I20260620 19:58:49.766742 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.740597 6903 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:49.736443 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.777040 8063 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:49.595125 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.788671 8055 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.585904 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.471112 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260620 19:58:49.796067 6225 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:49.594174 6377 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:49.593520 6374 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:49.592576 6228 raft_consensus.cc:2368] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7.
I20260620 19:58:49.591500 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.786168 8091 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:49.793762 8063 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:58:49.791106 6906 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:49.567798 6305 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:49.809604 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.633222 6907 raft_consensus.cc:2368] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:49.634511 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.468393 8056 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.791657 6380 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:49.812990 6230 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:49.564754 8045 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.813476 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:49.812191 8091 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.814249 6225 raft_consensus.cc:2393] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:49.799424 6198 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:49.782486 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:49.673436 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:49.590873 6875 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:49.815480 6381 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:58:49.663949 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.670077 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:49.817086 6198 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.818157 6198 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:49.816246 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 225411us
I20260620 19:58:49.467679 7995 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.539948 7976 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.819394 6875 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.810420 6228 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:49.558100 8040 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.467140 8023 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:49.564810 8080 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.564541 8042 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:49.813771 8045 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.695837 6379 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:58:49.671865 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.808177 6271 leader_election.cc:400] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7.
I20260620 19:58:49.595635 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.683264 6232 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:49.593874 6302 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:49.822818 6874 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:49.640733 7838 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.467355 7925 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:49.828168 6875 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:49.907299 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 79204.6us
I20260620 19:58:49.814996 8112 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:49.908241 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.642515 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.910053 8045 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.711738 6905 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:49.540277 8069 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.911279 8045 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:49.911743 8045 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.912426 6196 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.912506 8069 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.913177 8069 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:49.471177 7991 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.841414 8023 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.913674 8069 raft_consensus.cc:697] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:49.567931 8047 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.252s
I20260620 19:58:49.915061 7991 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.915205 8023 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.814994 8055 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:49.915683 8023 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:49.834935 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.905732 6271 leader_election.cc:403] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:58:49.905301 6376 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
W20260620 19:58:49.904980 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 82220.2us
I20260620 19:58:49.838042 8058 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.917394 8058 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.910140 7980 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.917779 8058 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:49.917847 7980 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.918341 7980 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:49.831931 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:49.917796 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.919953 8078 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.920084 6227 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:49.913281 6196 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.920856 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:49.469532 8038 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.921684 6196 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:49.922428 8047 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:49.917603 6874 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:49.916115 8023 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.922994 8038 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.923560 7991 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:49.923892 6298 raft_consensus.cc:2393] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:49.923986 7966 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.922955 8047 raft_consensus.cc:697] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:49.915982 8055 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.908123 8112 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:49.925061 8060 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:49.925529 8060 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.925398 8047 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:49.916038 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:49.925935 8060 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:49.923939 6377 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:49.926455 8055 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:49.927572 6232 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:49.816183 8056 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:49.827466 8099 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.956895 6381 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:58:49.832581 6373 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:49.956835 6196 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:49.827183 8092 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:49.824576 7968 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:49.924031 7991 raft_consensus.cc:697] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:49.990800 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.827029 8057 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260620 19:58:49.931200 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 123184us
I20260620 19:58:49.827713 8063 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:49.842114 8080 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:58:49.929914 8107 log.cc:927] Time spent T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20260620 19:58:49.901540 8042 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:49.990273 8092 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:50.023563 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:49.993597 7968 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:58:49.968741 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 411587us
I20260620 19:58:49.906733 7838 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.925056 8091 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:49.827348 8070 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:50.023010 8042 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:49.826084 7995 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:49.831905 8040 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:49.838168 7976 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.028406 7968 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:49.999398 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.001863 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.002723 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:49.965624 8078 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:49.978344 7933 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 7
I20260620 19:58:50.012899 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:49.967532 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.924196 7925 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:49.979135 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 516016us
I20260620 19:58:50.024955 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:49.926527 8112 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.116781 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.117229 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:50.026468 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.025377 6381 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.026475 6196 leader_election.cc:400] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:50.118803 6196 leader_election.cc:403] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:58:50.050632 7995 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.025530 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:50.119406 6301 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 5
I20260620 19:58:50.026705 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.123004 6299 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:50.031196 8063 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:50.125610 7995 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.126351 8063 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.127786 7995 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.128518 8063 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.015350 8080 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.006579 8057 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:50.041006 6198 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:50.044263 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.130095 8057 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.130553 6227 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:50.131424 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.131983 6227 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:50.132781 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.133271 6227 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:50.134007 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:49.956400 8071 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:50.134902 8071 raft_consensus.cc:697] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:50.135737 8071 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.137161 8057 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.011475 8045 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.022920 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:49.965807 8056 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.134502 6227 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:50.139621 8057 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.140157 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.025151 8092 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.142207 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.143061 6227 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:50.144098 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.145328 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:50.025529 8023 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:49.931988 8069 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.095077 6300 raft_consensus.cc:2368] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:50.040135 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.150808 6303 raft_consensus.cc:2393] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:50.041561 7838 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.097779 6226 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:50.115327 6305 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 4
W20260620 19:58:50.116351 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 96068.6us
I20260620 19:58:50.155339 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.156086 6303 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:50.157320 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:50.157949 6303 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.158627 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.160022 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.162004 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:50.179203 6299 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:50.191654 6303 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:50.048565 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:50.213122 6379 raft_consensus.cc:2393] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 2: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:50.046135 8070 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:50.214164 8070 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.206719 7968 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:50.193560 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 78577.9us
I20260620 19:58:50.219851 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:50.220474 6302 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.220947 6195 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:50.223416 6381 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:50.185731 6305 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:50.170423 7838 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.227300 6299 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:50.230981 6301 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:50.232676 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.233492 6305 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:50.232101 6302 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
W20260620 19:58:50.240495 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 215085us
I20260620 19:58:50.241348 6345 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:50.248234 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.261153 6300 raft_consensus.cc:2368] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:50.260041 8126 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:50.270700 8045 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:50.271096 8126 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.271250 8045 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.223129 8122 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:50.274380 6345 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:50.275002 6344 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:50.273280 8126 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.280437 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:58:50.281231 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:50.248188 8055 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:50.195370 8078 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.292433 6345 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:58:50.293311 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 266933us
I20260620 19:58:50.251626 8123 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:50.292681 6128 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47082 (request call id 406) took 3266 ms. Trace:
W20260620 19:58:50.294855 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 366550us
I20260620 19:58:50.282166 8057 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 4
W20260620 19:58:50.318727 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 277753us
I20260620 19:58:50.287528 8023 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:50.222430 7991 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.293083 8069 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:50.287153 8001 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:50.304502 6372 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:50.265221 8111 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260620 19:58:50.287691 8122 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.278362 8128 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:50.393647 6306 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:50.305863 7995 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:50.496037 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.508742 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:50.475256 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 233440us
I20260620 19:58:50.513079 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:50.513836 6910 raft_consensus.cc:2393] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:50.514099 6306 raft_consensus.cc:2393] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:50.513096 6271 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:50.514542 6273 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:50.511505 6307 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:50.317260 8091 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:50.531730 6273 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:50.323441 8118 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:50.545490 6874 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
W20260620 19:58:50.361697 8127 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260620 19:58:50.355301 6130 catalog_manager.cc:5696] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:50.502939 7976 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.510746 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:50.359583 7838 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:50.444657 8132 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:50.404544 8047 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:50.333896 8023 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:50.554376 6130 catalog_manager.cc:5696] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:50.555274 6306 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260620 19:58:50.555541 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 280564us
I20260620 19:58:50.510962 8044 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.557171 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:50.444514 8134 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:50.536186 8112 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260620 19:58:50.563082 8121 log.cc:927] Time spent T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
W20260620 19:58:50.574961 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 206961us
I20260620 19:58:50.511001 7995 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:50.532822 7925 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.545158 7965 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:50.545100 8118 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:50.545543 8139 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:50.551158 8079 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:50.570875 6307 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:50.511046 8001 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.528083 8092 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.573436 6130 catalog_manager.cc:5696] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 2 to 3, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 3 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:58:50.574277 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 249302us
I20260620 19:58:50.507992 8069 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:50.569990 6344 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:50.583261 8053 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:50.567181 8080 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.564888 6303 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.371608 8123 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:50.573612 7980 consensus_meta.cc:287] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.655s user 0.009s sys 0.000s
W20260620 19:58:50.585232 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 364326us
I20260620 19:58:50.569329 8041 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:50.589275 8118 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.528059 8128 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.572082 7976 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:50.595731 6198 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:50.595920 6130 catalog_manager.cc:5696] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:58:50.671370 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 284773us
I20260620 19:58:50.671622 6130 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 6 to 7, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 7 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:50.598040 6344 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:58:50.563264 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 270855us
I20260620 19:58:50.576330 8143 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:50.585759 7995 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:50.591965 8139 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:50.596242 8131 log.cc:927] Time spent T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20260620 19:58:50.571442 8023 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:50.659159 6195 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:50.568277 8132 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.661494 8041 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.612775 8147 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:50.598637 8069 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:50.590898 8079 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:50.662732 8080 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:50.602197 7965 raft_consensus.cc:697] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:50.305040 6128 rpcz_store.cc:276] 0620 19:58:47.013659 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:47.019135 (+ 5476us) service_pool.cc:224] Handling call
0620 19:58:50.280294 (+3261159us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:58:47.324259 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:58:47.325136 (+ 877us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:58:47.325199 (+ 63us) write_op.cc:435] Acquired schema lock
0620 19:58:47.325244 (+ 45us) tablet.cc:662] Decoding operations
0620 19:58:47.325744 (+ 500us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:58:47.325866 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0620 19:58:47.325908 (+ 42us) tablet.cc:685] Acquiring locks for 1 operations
0620 19:58:47.326279 (+ 371us) tablet.cc:701] Row locks acquired
0620 19:58:47.326321 (+ 42us) write_op.cc:260] PREPARE: finished
0620 19:58:47.326493 (+ 172us) write_op.cc:270] Start()
0620 19:58:47.326762 (+ 269us) write_op.cc:276] Timestamp: P: 1781985527326434 usec, L: 0
0620 19:58:47.326825 (+ 63us) op_driver.cc:348] REPLICATION: starting
0620 19:58:47.327569 (+ 744us) log.cc:844] Serialized 536 byte log entry
0620 19:58:48.622365 (+1294796us) op_driver.cc:464] REPLICATION: finished
0620 19:58:49.253519 (+631154us) write_op.cc:301] APPLY: starting
0620 19:58:49.265290 (+ 11771us) tablet.cc:1367] starting BulkCheckPresence
0620 19:58:49.283387 (+ 18097us) tablet.cc:1370] finished BulkCheckPresence
0620 19:58:49.284728 (+ 1341us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:58:49.285007 (+ 279us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:58:49.285264 (+ 257us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0620 19:58:49.285323 (+ 59us) write_op.cc:312] APPLY: finished
0620 19:58:49.285517 (+ 194us) log.cc:844] Serialized 23 byte log entry
0620 19:58:49.469727 (+184210us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:58:49.487973 (+ 18246us) write_op.cc:454] Released schema lock
0620 19:58:49.552959 (+ 64986us) write_op.cc:341] FINISH: Updating metrics
0620 19:58:49.656953 (+103994us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:58:49.664010 (+ 7057us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":589110,"apply.run_cpu_time_us":29301,"apply.run_wall_time_us":342507,"num_ops":1,"prepare.queue_time_us":227389,"prepare.run_cpu_time_us":34226,"prepare.run_wall_time_us":193261,"raft.queue_time_us":741084,"raft.run_cpu_time_us":112641,"raft.run_wall_time_us":484755,"replication_time_us":1295377,"thread_start_us":1153812,"threads_started":5,"wal-append.queue_time_us":752547,"wal-append.run_cpu_time_us":39599,"wal-append.run_wall_time_us":1534977}]]}
I20260620 19:58:50.664512 8123 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.578248 8056 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:50.674593 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 78954.3us
W20260620 19:58:50.675220 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 77233us
I20260620 19:58:50.595872 8047 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260620 19:58:50.660238 8053 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:50.678222 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:50.682248 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:50.685743 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 140317us
I20260620 19:58:50.686300 6195 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:50.714787 6378 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:58:50.684677 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 82076.7us
I20260620 19:58:50.673043 8078 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.676177 6130 catalog_manager.cc:5696] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:50.719295 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:50.719461 6195 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:50.680384 8111 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:50.705303 8079 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:50.580466 8134 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.677238 8148 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:58:50.670106 7980 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.674916 7976 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.701117 8045 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.668829 8142 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:50.721215 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:50.696427 7991 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 19:58:50.726450 8138 log.cc:927] Time spent T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260620 19:58:50.718463 8147 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:50.739053 8001 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.739368 8128 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.741405 8057 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:50.721345 8111 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:58:50.742811 6373 consensus_meta.cc:287] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.742s user 0.004s sys 0.007s
I20260620 19:58:50.758728 7976 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.758945 6373 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:50.759238 8139 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.759563 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1409) took 1199 ms. Trace:
I20260620 19:58:50.759934 6373 rpcz_store.cc:276] 0620 19:58:49.559714 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.560940 (+ 1226us) service_pool.cc:224] Handling call
0620 19:58:50.759481 (+1198541us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:50.744958 6909 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:50.760931 8092 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.744796 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 481) took 1643 ms. Trace:
I20260620 19:58:50.731655 7925 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:50.746006 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:50.727010 8091 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:50.730679 8122 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.748659 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:50.743449 8099 consensus_meta.cc:287] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.753s user 0.008s sys 0.003s
I20260620 19:58:50.765125 8041 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.735950 8045 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:50.737514 7966 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:50.756055 8053 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.761699 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1864) took 1306 ms. Trace:
W20260620 19:58:50.750337 7933 consensus_meta.cc:287] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.634s user 0.003s sys 0.007s
I20260620 19:58:50.721129 8078 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:50.762036 6298 rpcz_store.cc:276] 0620 19:58:49.093067 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.168609 (+ 75542us) service_pool.cc:224] Handling call
0620 19:58:50.730895 (+1562286us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b5400332554. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:50.736237 (+ 5342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1528366848}
I20260620 19:58:50.764490 8134 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:50.764424 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:50.680110 8143 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.749020 7991 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:50.813138 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1775) took 1245 ms. Trace:
I20260620 19:58:50.818491 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:50.811475 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1766) took 1246 ms. Trace:
I20260620 19:58:50.779081 6271 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:58:50.762039 8060 consensus_meta.cc:287] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.796s user 0.009s sys 0.000s
I20260620 19:58:50.753827 8123 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:50.780611 8118 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.780126 8091 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:50.819548 8135 log.cc:927] Time spent T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.580s user 0.003s sys 0.001s
I20260620 19:58:50.778019 8112 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:50.758767 8111 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:58:50.775151 8058 consensus_meta.cc:287] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.857s user 0.006s sys 0.006s
I20260620 19:58:50.742718 7965 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.779984 7925 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.802199 8001 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:50.829681 6378 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:50.796808 8147 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:50.790427 8099 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } }
I20260620 19:58:50.877121 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 259) took 1288 ms. Trace:
I20260620 19:58:50.859859 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:50.864344 6345 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:50.738116 8142 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.858695 6230 rpcz_store.cc:276] 0620 19:58:49.454963 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.455110 (+ 147us) service_pool.cc:224] Handling call
0620 19:58:50.761537 (+1306427us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54000fb4d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:50.761626 (+ 89us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":564977024}
W20260620 19:58:50.835011 8040 consensus_meta.cc:287] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.785s user 0.003s sys 0.007s
I20260620 19:58:50.798400 6303 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
W20260620 19:58:50.820559 8042 consensus_meta.cc:287] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.760s user 0.008s sys 0.000s
I20260620 19:58:50.857801 6232 rpcz_store.cc:276] 0620 19:58:49.567280 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.907945 (+340665us) service_pool.cc:224] Handling call
0620 19:58:50.812990 (+905045us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b54000edb54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:50.813086 (+ 96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42824064}
W20260620 19:58:50.861174 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 82135.7us
I20260620 19:58:50.860080 6227 rpcz_store.cc:276] 0620 19:58:49.564550 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.146379 (+581829us) service_pool.cc:224] Handling call
0620 19:58:50.186478 (+ 40099us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:50.248280 (+ 61802us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:50.248312 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:50.263828 (+ 15516us) log.cc:844] Serialized 26 byte log entry
0620 19:58:50.565428 (+301600us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:50.565482 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:50.565564 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:50.811243 (+245679us) raft_consensus.cc:1685] finished
0620 19:58:50.811270 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:50.811406 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59443,"raft.queue_time_us":116176,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":43533,"thread_start_us":265126,"threads_started":1,"wal-append.queue_time_us":387914}
I20260620 19:58:50.985549 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:50.805756 6372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1411) took 1245 ms. Trace:
I20260620 19:58:50.884034 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:58:50.883729 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 164311us
I20260620 19:58:51.026909 6376 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.802819 8132 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:51.026052 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 161775us
W20260620 19:58:50.968613 8070 consensus_meta.cc:287] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.754s user 0.011s sys 0.000s
I20260620 19:58:51.027590 6271 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:51.028432 8080 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:50.851563 8056 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.031031 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:50.875589 7966 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.877964 8050 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:51.030553 6225 rpcz_store.cc:276] 0620 19:58:49.588309 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.582487 (+994178us) service_pool.cc:224] Handling call
0620 19:58:50.592453 (+ 9966us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:50.662580 (+ 70127us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:50.662627 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:50.662666 (+ 39us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:50.662694 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:50.876849 (+214155us) spinlock_profiling.cc:243] Waited 70 ms on lock 0x7b54000b4d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:50.876907 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:50.877055 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":160897280}
W20260620 19:58:50.945784 8158 log.cc:927] Time spent T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260620 19:58:50.875810 7933 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7.
I20260620 19:58:51.052762 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.036029 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.065485 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:51.075001 6227 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:51.003518 7980 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.076737 8146 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:51.031105 6195 leader_election.cc:400] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:51.032912 8091 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:50.990926 8118 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:51.068344 8050 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:51.073653 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.030282 6372 rpcz_store.cc:276] 0620 19:58:49.560351 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.560976 (+ 625us) service_pool.cc:224] Handling call
0620 19:58:50.805507 (+1244531us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54000456d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:50.805683 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":827125632}
W20260620 19:58:51.077584 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 49954.3us
I20260620 19:58:51.149067 6225 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 4
I20260620 19:58:50.885125 6379 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:51.067122 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:51.075018 6378 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:51.034739 6373 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:51.079345 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.064642 8001 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.032721 8023 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.070027 6223 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:51.066148 8145 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:50.877321 8078 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:50.876962 8060 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.208508 8078 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.208576 8060 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:50.882905 8148 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:50.883876 8123 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.070569 7991 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:51.149511 6195 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:51.210678 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 61223.4us
I20260620 19:58:51.210259 6875 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.212054 6195 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.213871 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.034467 8040 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.215518 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.100809 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.216651 6227 raft_consensus.cc:2368] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3.
W20260620 19:58:51.154834 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 87779.2us
I20260620 19:58:51.217273 6875 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.033339 8042 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.050884 8070 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.155005 7933 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:51.229311 6344 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:51.160769 6376 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:51.163229 6230 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:51.028276 8058 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.141130 8080 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:51.150468 8050 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.193863 6378 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:51.176659 6225 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:51.235015 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.150215 8069 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.209693 8148 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.214879 8056 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:51.220414 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:51.223037 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:51.241253 6306 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.222970 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.223507 6875 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.224364 6375 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:51.225589 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.216485 8060 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.236413 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:51.232903 6271 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.129:36725 timed out after 1.658s (SENT) [suppressed 3 similar messages]
I20260620 19:58:51.237577 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.237643 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.249560 6307 raft_consensus.cc:2393] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:51.227497 7980 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:51.244602 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.206212 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.244320 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.255475 6230 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:51.245363 6905 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:51.245760 6875 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.235255 8091 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:51.262509 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.224614 8143 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.223809 7991 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.249619 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.222667 8128 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.251327 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.228662 8132 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:51.231506 8078 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:51.229785 8052 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:51.245498 6231 raft_consensus.cc:2393] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:51.277269 6908 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:58:51.147639 8079 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.278258 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.263522 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.245654 8056 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:51.270174 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.279783 6225 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:51.244302 6227 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
W20260620 19:58:51.259717 7995 consensus_meta.cc:287] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.574s user 0.007s sys 0.003s
I20260620 19:58:51.273514 6875 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.250871 8156 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:51.251746 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:51.252544 7980 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:51.275967 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.257275 8040 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.281729 8052 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.281805 8165 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:58:51.282429 8167 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20260620 19:58:51.283308 8023 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.284584 8079 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.272238 8155 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:51.267238 6372 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:51.288761 8069 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.288888 8056 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.261307 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.244807 8080 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.290071 8132 raft_consensus.cc:697] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:51.290271 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.326413 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.326782 6230 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.327235 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.290724 8078 raft_consensus.cc:697] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:51.282545 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.293061 7925 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.307039 8126 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:51.294994 8001 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.295351 6304 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:58:51.268646 6306 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:51.260147 8144 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:58:51.298898 8091 raft_consensus.cc:697] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:51.301358 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.309767 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.320037 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.340415 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.274948 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.256036 8060 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.305303 8156 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:51.331266 6379 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:51.313410 7980 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.320593 6298 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:51.319331 8052 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:51.338732 6225 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:51.371409 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.371960 6225 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.376952 7968 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.379756 6225 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:51.380693 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.381217 6225 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:58:51.382164 6223 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1866) took 1924 ms (client timeout 2214 ms). Trace:
I20260620 19:58:51.389688 6225 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:51.391047 6195 leader_election.cc:400] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:51.390949 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.391485 6195 leader_election.cc:403] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:58:51.300045 8128 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.322942 8155 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:51.307343 7995 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.340762 6376 raft_consensus.cc:2376] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4. Re-sending same reply.
W20260620 19:58:51.328969 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 96078us
I20260620 19:58:51.346692 6230 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:51.347065 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.347416 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.338960 8001 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.349586 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 504) took 1420 ms. Trace:
I20260620 19:58:51.349858 8156 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:51.344748 8144 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:51.331240 8126 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:51.351063 8132 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.354382 8056 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:51.419238 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 107925us
I20260620 19:58:51.420158 8142 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.420433 8056 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.357956 6226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1767) took 1792 ms. Trace:
I20260620 19:58:51.420787 8091 consensus_queue.cc:237] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:51.420176 6877 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:51.421121 6226 rpcz_store.cc:276] 0620 19:58:49.565028 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.152132 (+587104us) service_pool.cc:224] Handling call
0620 19:58:50.186560 (+ 34428us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:50.247880 (+ 61320us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:50.247909 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:50.262924 (+ 15015us) log.cc:844] Serialized 26 byte log entry
0620 19:58:50.677514 (+414590us) raft_consensus.cc:1645] Marking committed up to 1
0620 19:58:50.677579 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.357587 (+680008us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b54000fb9d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:51.357629 (+ 42us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:51.357671 (+ 42us) raft_consensus.cc:1685] finished
0620 19:58:51.357684 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.357873 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26398,"raft.queue_time_us":111998,"spinlock_wait_cycles":90289536,"thread_start_us":397228,"threads_started":1,"wal-append.queue_time_us":501312}
I20260620 19:58:51.422984 8053 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:58:51.423275 6227 raft_consensus.cc:2393] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:51.362208 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.422993 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.424197 6226 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.424630 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.425174 6377 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.364321 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.359160 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 501) took 1449 ms. Trace:
I20260620 19:58:51.361344 6910 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:51.322546 7966 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.426767 8050 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.426789 6909 rpcz_store.cc:276] 0620 19:58:49.909616 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.920642 (+ 11026us) service_pool.cc:224] Handling call
0620 19:58:49.920888 (+ 246us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:50.761158 (+840270us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.358940 (+597782us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b54005e15d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:51.359085 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1894564736}
I20260620 19:58:51.428316 7966 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:51.428438 8070 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:51.354576 8159 log.cc:927] Time spent T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.502s user 0.002s sys 0.000s
I20260620 19:58:51.430074 8080 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:51.382736 6223 rpcz_store.cc:269] 0620 19:58:49.457554 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.461758 (+ 4204us) service_pool.cc:224] Handling call
0620 19:58:49.965502 (+503744us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5800000c98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57916ee 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:51.381974 (+1416472us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b54000fa5d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:51.382106 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2141242624}
I20260620 19:58:51.430692 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:51.415454 8078 consensus_queue.cc:237] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.431273 6903 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.417510 8126 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.432636 6226 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:51.418092 8144 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:58:51.433501 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.434257 6303 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:51.361418 8060 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.325707 8165 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.405537 6907 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:51.434522 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.436662 7968 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:51.437162 7968 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.437541 6226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 264) took 1782 ms. Trace:
I20260620 19:58:51.437889 6226 rpcz_store.cc:276] 0620 19:58:49.655502 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.436858 (+1781356us) service_pool.cc:224] Handling call
0620 19:58:51.437103 (+ 245us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:51.437266 (+ 163us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:51.437291 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:51.437331 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:51.437361 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.437412 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.437498 (+ 86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:51.427392 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.439416 6902 raft_consensus.cc:1240] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:51.439715 6877 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:51.439883 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.440397 6381 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:51.440558 6374 raft_consensus.cc:2368] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5.
I20260620 19:58:51.409714 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.442008 6225 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:51.411020 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.442988 6376 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:51.444053 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.444970 6376 raft_consensus.cc:2393] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:51.361725 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.450208 6226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1781) took 1769 ms. Trace:
I20260620 19:58:51.450548 6372 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:51.451153 8052 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.451498 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.451627 8111 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.452349 6372 raft_consensus.cc:2393] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:51.452762 6378 raft_consensus.cc:2368] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:58:51.453143 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.453507 6877 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:51.454010 6229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 256) took 1867 ms. Trace:
I20260620 19:58:51.359452 8058 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.362251 8123 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.431175 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.445079 6344 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:51.469826 8128 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.448381 7966 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:51.497066 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.461095 6877 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:51.461328 6271 leader_election.cc:400] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3.
I20260620 19:58:51.463083 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.498127 6271 leader_election.cc:403] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:58:51.498683 6306 raft_consensus.cc:2393] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:51.468101 6377 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:51.454376 6229 rpcz_store.cc:276] 0620 19:58:49.586393 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.545577 (+959184us) service_pool.cc:224] Handling call
0620 19:58:51.453326 (+907749us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b5800000c98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57916ee 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:58:51.453479 (+ 153us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:51.453667 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:51.453694 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:51.453741 (+ 47us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:51.453802 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.453859 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.453958 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6178176}
I20260620 19:58:51.469602 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.500124 6271 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.500626 6225 raft_consensus.cc:2393] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:51.404556 8164 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:51.501402 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.412290 8079 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:51.474074 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.474388 6903 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:51.450583 6226 rpcz_store.cc:276] 0620 19:58:49.681231 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.449499 (+1768268us) service_pool.cc:224] Handling call
0620 19:58:51.449780 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:51.449936 (+ 156us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:51.449957 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:51.449996 (+ 39us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:58:51.450024 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.450071 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.450160 (+ 89us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:51.453742 6372 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.435109 7991 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.595155 6300 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260620 19:58:51.406813 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.404735 8155 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:51.412539 8001 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.453739 8111 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:51.503595 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:51.485358 8058 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:51.504243 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.409317 6906 rpcz_store.cc:276] 0620 19:58:49.928919 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.933374 (+ 4455us) service_pool.cc:224] Handling call
0620 19:58:50.813438 (+880064us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:51.309632 (+496194us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b579169c 00007fa6b579125b 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:51.349506 (+ 39874us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29951616}
I20260620 19:58:51.505167 8126 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.507099 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:51.506788 7980 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.507350 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.642123 6225 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 6
I20260620 19:58:51.645750 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.647032 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.647536 6378 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.507804 8125 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.521564 6381 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:51.516606 6875 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:51.484843 8080 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:51.499326 8123 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:51.595244 6372 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:51.609721 8052 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:51.503472 8128 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:51.611836 6306 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:51.518699 8145 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
W20260620 19:58:51.615223 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 245806us
I20260620 19:58:51.620173 6375 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:51.707186 6377 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:51.613088 6909 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:51.582756 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 710) took 1308 ms. Trace:
I20260620 19:58:51.579852 8079 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.642117 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1780) took 2056 ms. Trace:
I20260620 19:58:51.608879 7968 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.625468 8023 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.662519 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.643256 8118 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.664999 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.701278 7933 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:51.700513 8041 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:58:51.700328 6231 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1773) took 2133 ms (client timeout 1658 ms). Trace:
I20260620 19:58:51.612171 8050 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.689579 6225 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:58:51.715718 6198 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.718043 6224 rpcz_store.cc:276] 0620 19:58:49.586147 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.529383 (+943236us) service_pool.cc:224] Handling call
0620 19:58:50.529745 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:50.736346 (+206601us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:50.736374 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:50.736414 (+ 40us) raft_consensus.cc:1645] Marking committed up to 1
0620 19:58:50.736442 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.641898 (+905456us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b5400286254. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:51.641946 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.642067 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":474701568}
I20260620 19:58:51.719995 6907 raft_consensus.cc:2368] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:51.661271 6378 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:51.720875 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.703581 8111 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.741278 6375 raft_consensus.cc:2393] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:51.721326 6229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 268) took 1009 ms. Trace:
I20260620 19:58:51.741943 6229 rpcz_store.cc:276] 0620 19:58:50.711473 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.505556 (+794083us) service_pool.cc:224] Handling call
0620 19:58:51.505791 (+ 235us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:51.505964 (+ 173us) raft_consensus.cc:1534] Early marking committed up to index 5
0620 19:58:51.506221 (+ 257us) log.cc:844] Serialized 14 byte log entry
0620 19:58:51.720922 (+214701us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:51.720990 (+ 68us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:51.721044 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.721128 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.721263 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1472,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":106,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3422,"thread_start_us":214292,"threads_started":1}
I20260620 19:58:51.731367 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.721668 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.718220 8080 raft_consensus.cc:697] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:51.662165 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.735430 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:51.735606 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 221581us
I20260620 19:58:51.750888 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:51.751276 6305 raft_consensus.cc:1240] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:51.751420 6301 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:51.704602 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:51.753137 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.753384 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.705000 8146 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:58:51.737288 6231 rpcz_store.cc:269] 0620 19:58:49.566467 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.640960 (+ 74493us) service_pool.cc:224] Handling call
0620 19:58:51.700098 (+2059138us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001d56d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:51.700254 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3027234048}
I20260620 19:58:51.745868 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.745738 6381 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:51.755609 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:51.757063 6198 leader_election.cc:400] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:58:51.757584 6198 leader_election.cc:403] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:58:51.733253 8145 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:51.758157 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.717896 8123 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.765307 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:58:51.733304 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 77622.4us
I20260620 19:58:51.759881 6301 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:58:51.746658 8132 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:51.771240 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.772349 6198 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.770041 6375 raft_consensus.cc:2368] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:51.773640 6198 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.776376 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 342) took 1095 ms. Trace:
I20260620 19:58:51.777724 6198 leader_election.cc:400] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:51.778159 6198 leader_election.cc:403] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260620 19:58:51.734781 8023 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.777041 6376 rpcz_store.cc:276] 0620 19:58:50.681159 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.458591 (+777432us) service_pool.cc:224] Handling call
0620 19:58:51.458904 (+ 313us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:51.459121 (+ 217us) raft_consensus.cc:1534] Early marking committed up to index 5
0620 19:58:51.483094 (+ 23973us) log.cc:844] Serialized 14 byte log entry
0620 19:58:51.775492 (+292398us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:51.775594 (+ 102us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:51.775657 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.775741 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.776270 (+ 529us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25263,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":8368,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":2691,"thread_start_us":291985,"threads_started":2}
I20260620 19:58:51.717804 6903 rpcz_store.cc:276] 0620 19:58:50.274586 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.277112 (+ 2526us) service_pool.cc:224] Handling call
0620 19:58:51.430560 (+1153448us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57916ee 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:51.582684 (+152124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6577408}
I20260620 19:58:51.785496 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.787536 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.767959 8148 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:58:51.797174 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.741959 8041 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.741060 8118 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.784448 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:51.796063 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:51.786127 6373 consensus_meta.cc:287] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.587s user 0.011s sys 0.000s
I20260620 19:58:51.805534 6225 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:51.795296 6874 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:51.765021 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.797925 6910 raft_consensus.cc:2393] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:51.742204 7933 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.799062 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:51.752691 8111 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.757133 8053 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260620 19:58:51.771359 8132 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:51.780694 7991 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:51.780884 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:51.782452 8143 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:51.785835 8154 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:51.811265 6302 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:51.820565 6906 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:51.820827 8191 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260620 19:58:51.821542 8180 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260620 19:58:51.821934 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 332) took 1144 ms. Trace:
I20260620 19:58:51.822702 6228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 258) took 2235 ms. Trace:
I20260620 19:58:51.808372 6373 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:51.762830 8146 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.815637 8143 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.823055 6228 rpcz_store.cc:276] 0620 19:58:49.587971 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.562031 (+974060us) service_pool.cc:224] Handling call
0620 19:58:51.355654 (+793623us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b5800000c98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:58:51.355873 (+ 219us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:51.356253 (+ 380us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:51.356292 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:51.356813 (+ 521us) log.cc:844] Serialized 26 byte log entry
0620 19:58:51.651654 (+294841us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:58:51.651692 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.651753 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:51.822395 (+170642us) raft_consensus.cc:1685] finished
0620 19:58:51.822439 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.822648 (+ 209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10425,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":16279,"spinlock_wait_cycles":2587392,"thread_start_us":294517,"threads_started":1,"wal-append.queue_time_us":360371}
I20260620 19:58:51.816638 8132 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:51.842265 6376 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:51.828671 6372 raft_consensus.cc:2368] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
W20260620 19:58:51.834812 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 165698us
I20260620 19:58:51.824003 8053 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:51.842054 8126 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:51.829778 7991 raft_consensus.cc:697] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:51.826304 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.870165 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:51.895327 6228 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:51.896965 6230 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:51.898855 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:51.900568 8183 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260620 19:58:51.901347 8156 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:51.902065 8187 log.cc:927] Time spent T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260620 19:58:51.903988 8123 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:51.905758 6298 consensus_meta.cc:287] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.538s user 0.006s sys 0.002s
I20260620 19:58:51.809016 8080 consensus_queue.cc:237] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.828375 8145 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.906629 8001 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:51.886178 8186 log.cc:927] Time spent T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260620 19:58:51.886762 7933 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:51.886962 8188 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260620 19:58:51.888211 8099 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:51.840292 8176 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:51.855243 8143 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:58:51.855435 8175 log.cc:927] Time spent T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20260620 19:58:51.893661 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.929109 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:51.930490 6908 raft_consensus.cc:1240] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:51.906097 6298 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:51.868791 6345 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:51.851869 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 345) took 1166 ms. Trace:
I20260620 19:58:51.933389 6198 leader_election.cc:400] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:51.945084 6198 leader_election.cc:403] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260620 19:58:51.891546 8147 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:51.892737 8091 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:58:51.946897 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 78199.9us
I20260620 19:58:51.950114 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.950095 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:51.950942 6225 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:51.953104 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:51.931067 8079 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:51.940951 8144 consensus_meta.cc:287] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.507s user 0.010s sys 0.000s
I20260620 19:58:51.954813 8197 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:51.944293 6228 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:51.950587 8099 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:51.950557 8041 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.959268 6345 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.954083 8023 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:51.950824 8196 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:51.945024 6373 rpcz_store.cc:276] 0620 19:58:50.681841 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.765224 (+ 83383us) service_pool.cc:224] Handling call
0620 19:58:51.848198 (+1082974us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:51.964092 7991 consensus_queue.cc:237] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.980108 8118 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:51.961601 8197 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:51.952562 8147 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:51.991516 8156 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:51.988649 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 472058us
I20260620 19:58:51.954521 8091 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:51.973860 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:51.963860 6905 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 505) took 2033 ms (client timeout 1778 ms). Trace:
I20260620 19:58:51.994436 6228 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:51.971868 8146 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:51.986699 8023 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:51.961964 8144 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.003130 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:52.004665 6196 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.006008 6196 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.006438 6225 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:52.007387 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:52.007871 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:52.008415 6231 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:51.988781 8197 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.010200 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:52.008183 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:52.009459 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 214233us
I20260620 19:58:52.012903 6230 raft_consensus.cc:2393] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:52.012917 6225 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:52.014003 6196 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.016840 6196 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.017450 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:52.012066 8023 raft_consensus.cc:697] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:52.020553 6228 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:52.012629 8154 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:52.015476 8123 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:52.019439 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:52.023723 6379 consensus_meta.cc:287] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.619s user 0.011s sys 0.000s
I20260620 19:58:52.021358 8070 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:52.000191 8099 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
W20260620 19:58:52.044127 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 376583us
I20260620 19:58:52.018947 8164 consensus_queue.cc:1059] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260620 19:58:52.013173 8091 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.092777 6273 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:52.093365 8196 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.093550 7925 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:58:52.096284 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 72498.2us
W20260620 19:58:52.099516 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 140261us
I20260620 19:58:52.104132 6345 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:52.043848 8125 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3.
I20260620 19:58:52.105232 6345 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:51.822314 6380 rpcz_store.cc:276] 0620 19:58:50.677574 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.682227 (+ 4653us) service_pool.cc:224] Handling call
0620 19:58:50.966801 (+284574us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b5800142e98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57916ee 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:58:50.966981 (+ 180us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:50.967188 (+ 207us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:58:51.203770 (+236582us) log.cc:844] Serialized 14 byte log entry
0620 19:58:51.449946 (+246176us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b1800370300. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b84595da 00007fa6b88c4e0a
0620 19:58:51.802048 (+352102us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:51.808161 (+ 6113us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:58:51.810108 (+ 1947us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:51.812361 (+ 2253us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:51.821849 (+ 9488us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":281738,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":1700,"raft.queue_time_us":17951,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":5598,"spinlock_wait_cycles":25585152,"thread_start_us":504102,"threads_started":3,"wal-append.queue_time_us":517793}
I20260620 19:58:52.111514 6345 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:52.106048 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 89252.7us
I20260620 19:58:52.108690 6195 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.015383 8133 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
W20260620 19:58:52.043998 6304 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1379) took 2040 ms (client timeout 2033 ms). Trace:
W20260620 19:58:52.124868 8198 log.cc:927] Time spent T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20260620 19:58:52.038991 8202 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.124256 8206 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.127382 6307 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
W20260620 19:58:51.995066 6905 rpcz_store.cc:269] 0620 19:58:49.929299 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:49.937697 (+ 8398us) service_pool.cc:224] Handling call
0620 19:58:50.345791 (+408094us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57916ee 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:50.748561 (+402770us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b579178c 00007fa6b57916f6 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:51.955133 (+1206572us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54004b8054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:51.963105 (+ 7972us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1003912576}
I20260620 19:58:52.129770 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.129856 6195 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:52.138289 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:58:52.183748 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 53934.5us
I20260620 19:58:52.184700 6195 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.043521 6379 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:52.130929 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.132339 6306 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:52.187212 6305 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:52.044387 8042 leader_election.cc:290] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.135684 6196 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:52.103057 7991 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:52.095155 8070 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:52.103070 8091 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:52.092355 6373 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:52.110945 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:52.175239 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 71192.1us
I20260620 19:58:52.181954 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.093067 8146 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.126484 8052 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:52.192097 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:52.263177 6378 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:52.232913 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1420) took 1519 ms. Trace:
W20260620 19:58:52.232997 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 121505us
I20260620 19:58:52.194619 6301 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:52.248795 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.264201 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:52.118474 8143 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.250764 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.265856 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.245791 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.246361 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.267066 6300 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:52.131237 8164 raft_consensus.cc:3055] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
W20260620 19:58:52.272640 8209 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
I20260620 19:58:52.272713 8164 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:52.272996 6299 raft_consensus.cc:2393] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:52.273859 8215 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:52.273655 8164 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.274958 8164 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.275347 8144 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.263811 6377 rpcz_store.cc:276] 0620 19:58:50.713208 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.507737 (+794529us) service_pool.cc:224] Handling call
0620 19:58:51.707025 (+199288us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:51.707535 (+ 510us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:52.193638 (+486103us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b5400044cd0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:52.232850 (+ 39212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":457642880}
I20260620 19:58:52.234102 6375 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:52.270543 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.041034 8023 consensus_queue.cc:237] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.242434 8070 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.271885 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.121003 8079 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:52.271270 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.276705 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:52.289093 6377 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:52.290200 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:52.290967 6377 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:52.253468 8042 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:52.256485 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:52.275949 8144 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.118703 7925 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.295671 6273 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:52.246824 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 348) took 1548 ms. Trace:
I20260620 19:58:52.288115 6306 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:52.286080 6301 raft_consensus.cc:2376] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7. Re-sending same reply.
I20260620 19:58:52.297648 6904 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:52.325569 6273 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:52.331701 6306 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:52.337247 8205 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:52.337661 8205 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:52.337989 8205 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:52.338282 6306 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:58:52.255851 8206 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.340889 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.340780 8197 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.327958 6301 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260620 19:58:52.276911 8210 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:52.289647 6874 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:52.178746 8212 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:58:52.261090 8080 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:52.276527 7991 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:52.261399 8146 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:52.335203 8155 consensus_meta.cc:287] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.702s user 0.009s sys 0.002s
I20260620 19:58:52.296525 8053 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260620 19:58:52.298715 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 114074us
I20260620 19:58:52.282243 8052 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.355916 6195 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:52.359434 7980 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:52.321861 8079 raft_consensus.cc:697] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:58:52.348335 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 58748.8us
W20260620 19:58:52.176136 6304 rpcz_store.cc:269] 0620 19:58:50.003282 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.010977 (+ 7695us) service_pool.cc:224] Handling call
0620 19:58:52.043795 (+2032818us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54001cf054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:52.043925 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2307964160}
I20260620 19:58:52.366427 8099 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:52.296758 8207 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:52.366894 8099 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:52.367312 8099 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.367141 8207 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.346822 8147 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:52.352161 8156 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:58:52.362848 6195 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:52.256548 8091 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.327904 6380 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:52.354117 8080 raft_consensus.cc:697] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:52.350991 6231 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:52.374235 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.383562 6301 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:52.328490 8144 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.125890 8196 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.376528 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.381309 6195 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.322042 8193 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.190032 8202 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.356907 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.326794 6379 rpcz_store.cc:276] 0620 19:58:50.698049 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.236625 (+538576us) service_pool.cc:224] Handling call
0620 19:58:52.232706 (+996081us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:52.390797 6195 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.391561 8156 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.393972 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:52.384609 8164 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:52.396572 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1782) took 1003 ms. Trace:
I20260620 19:58:52.396989 6380 rpcz_store.cc:276] 0620 19:58:51.392930 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.391638 (+998708us) service_pool.cc:224] Handling call
0620 19:58:52.391948 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:52.396207 (+ 4259us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:52.396247 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:52.396306 (+ 59us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:58:52.396355 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:52.396416 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:52.396518 (+ 102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:52.393271 8206 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:52.398697 7925 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.398867 8133 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:52.399564 8133 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:52.399940 8133 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:52.400270 8133 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:52.400482 8164 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:52.386515 8193 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.361521 8155 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.377148 8053 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260620 19:58:52.373234 7991 consensus_queue.cc:1059] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260620 19:58:52.376734 8042 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260620 19:58:52.407819 8156 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.417881 8193 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:52.389394 8147 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:52.422066 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.402649 8212 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260620 19:58:52.427096 6908 raft_consensus.cc:1240] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:52.394835 8216 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:52.398622 8214 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 19:58:52.420147 8132 consensus_meta.cc:287] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.529s user 0.010s sys 0.000s
W20260620 19:58:52.471694 6346 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.000s sys 0.025s
W20260620 19:58:52.472558 6346 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.514s user 0.000s sys 0.025s
I20260620 19:58:52.408424 7995 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.421676 8125 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:52.419430 7991 raft_consensus.cc:3055] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
W20260620 19:58:52.472184 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 207947us
I20260620 19:58:52.427452 8147 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:52.425889 8215 raft_consensus.cc:697] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:52.505765 6345 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:52.419296 8079 consensus_queue.cc:237] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.519462 8144 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.521104 8144 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:52.521662 8144 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.529071 8070 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.532326 8196 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.533010 8196 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:58:52.533509 8196 raft_consensus.cc:697] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:52.538937 8156 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:52.431559 8197 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:52.543845 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 408048us
I20260620 19:58:52.544953 6196 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:52.475415 7991 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:52.507278 8147 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:52.487401 8143 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:52.552799 8176 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:52.553236 8176 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:52.554996 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1886) took 1314 ms. Trace:
I20260620 19:58:52.554801 8215 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.555373 6227 rpcz_store.cc:276] 0620 19:58:51.240353 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.282314 (+ 41961us) service_pool.cc:224] Handling call
0620 19:58:52.554805 (+1272491us) spinlock_profiling.cc:243] Waited 94.9 ms on lock 0x7b540004c4d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:52.554932 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":218298112}
I20260620 19:58:52.559060 7980 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.559574 7980 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.472796 8080 consensus_queue.cc:237] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.475184 8132 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.548878 8212 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:52.562250 6196 leader_election.cc:400] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:52.534484 8196 consensus_queue.cc:237] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.561470 7980 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:52.554256 8099 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.580940 6877 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:52.549975 8214 consensus_queue.cc:1059] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260620 19:58:52.568218 8212 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:52.578713 8052 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:52.591773 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 52495.6us
I20260620 19:58:52.583403 8202 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:52.583426 8210 raft_consensus.cc:697] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:52.583060 8193 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.583730 8091 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:52.588985 8206 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election won for term 3
W20260620 19:58:52.597611 6303 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1648) took 1379 ms (client timeout 1660 ms). Trace:
I20260620 19:58:52.590662 8212 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:58:52.603526 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 86904.8us
W20260620 19:58:52.597935 6303 rpcz_store.cc:269] 0620 19:58:51.218718 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.220124 (+ 1406us) service_pool.cc:224] Handling call
0620 19:58:52.597434 (+1377310us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54003246d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:52.597568 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444169088}
I20260620 19:58:52.615302 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.617038 6303 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:58:52.585330 7980 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260620 19:58:52.595113 8155 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:52.611279 6345 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:52.603777 8070 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.626842 6345 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:52.630576 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:52.634485 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:52.635195 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.635318 8220 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:52.635989 6904 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:52.637117 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:52.638343 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:52.638769 6345 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:52.639549 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.640017 7925 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:52.635998 8220 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.640085 6908 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:52.610044 8206 raft_consensus.cc:697] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:52.644835 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:52.642741 6345 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:52.655164 6231 raft_consensus.cc:1275] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:52.655364 6345 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:52.657079 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:52.658937 8219 log.cc:927] Time spent T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260620 19:58:52.659231 8156 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:52.659376 8222 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:52.659830 8156 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.660231 8222 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:52.660284 8156 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:52.641424 8221 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:52.661197 8221 raft_consensus.cc:697] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:52.661752 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:52.646960 8023 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:52.663673 8023 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.663861 6345 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:52.664047 8023 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.659673 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:52.665750 6877 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:52.666958 6877 leader_election.cc:304] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:52.671720 6231 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:52.674479 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:52.675446 6231 raft_consensus.cc:2393] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:52.678364 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.621629 8216 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260620 19:58:52.646347 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:52.619746 8193 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.646272 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:52.619552 8215 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
I20260620 19:58:52.645985 8144 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.662070 8221 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.660346 8132 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.678073 8202 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:52.664790 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.620605 8133 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.680732 8222 consensus_queue.cc:1059] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260620 19:58:52.719676 8157 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:52.720692 6307 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1654) took 1260 ms (client timeout 1621 ms). Trace:
W20260620 19:58:52.728345 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 52201.7us
W20260620 19:58:52.729154 6307 rpcz_store.cc:269] 0620 19:58:51.460249 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.460972 (+ 723us) service_pool.cc:224] Handling call
0620 19:58:52.720523 (+1259551us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b54003241d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:52.720634 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1400661888}
I20260620 19:58:52.684623 7925 consensus_queue.cc:1059] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260620 19:58:52.697451 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:52.696774 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.685070 8091 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:52.695541 8165 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.677623 8052 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260620 19:58:52.722704 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 58887.2us
I20260620 19:58:52.735596 6227 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:58:52.659133 8206 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.683826 8196 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260620 19:58:52.739705 6231 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:52.723038 8202 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.689980 8041 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:52.734779 6905 raft_consensus.cc:2393] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:52.732313 6902 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:52.734719 8230 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:58:52.749888 8220 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.733747 8091 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.735167 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.683399 8210 consensus_queue.cc:237] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.755756 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.853992 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:52.853008 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 70406us
I20260620 19:58:52.742816 8052 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:52.856323 6345 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:52.752805 8202 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:52.860332 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:58:52.856942 8205 consensus_meta.cc:287] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.519s user 0.011s sys 0.001s
I20260620 19:58:52.872409 6306 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:52.871338 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:52.873720 6307 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.855289 8238 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:52.873247 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.789680 8099 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.859899 8091 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.874861 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:52.789100 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1431) took 1122 ms. Trace:
W20260620 19:58:52.877496 6374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1424) took 2159 ms (client timeout 1826 ms). Trace:
I20260620 19:58:52.857074 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:52.748339 8157 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.855811 8222 consensus_queue.cc:1048] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260620 19:58:52.865934 8132 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:58:52.881003 6374 rpcz_store.cc:269] 0620 19:58:50.714378 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:50.985236 (+270858us) service_pool.cc:224] Handling call
0620 19:58:52.868960 (+1883724us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54000d4dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:52.873444 (+ 4484us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":952398720}
I20260620 19:58:52.868062 8052 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:52.793434 8041 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.864912 8220 consensus_queue.cc:1059] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260620 19:58:52.890271 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.889415 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.899844 8125 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:52.900357 8125 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:52.881507 8238 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:52.873606 8205 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.922662 6301 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:58:52.925186 8193 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:52.931459 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.932121 6373 raft_consensus.cc:2376] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4. Re-sending same reply.
I20260620 19:58:52.933034 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.936579 8238 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:52.937413 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:52.938159 6231 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:52.938625 8157 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:52.942902 8176 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.945266 8176 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:52.946201 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:52.946962 6381 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:52.948014 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.948303 8147 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.949326 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.949987 6381 raft_consensus.cc:2376] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4. Re-sending same reply.
I20260620 19:58:52.939332 6345 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:52.939531 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.951778 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 354) took 1641 ms. Trace:
I20260620 19:58:52.952387 6376 rpcz_store.cc:276] 0620 19:58:51.310104 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.806004 (+495900us) service_pool.cc:224] Handling call
0620 19:58:51.839811 (+ 33807us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:51.854732 (+ 14921us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:52.951603 (+1096871us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b54000eb850. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:52.951725 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77138176}
I20260620 19:58:52.952960 7995 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:52.953574 7995 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:52.953929 7995 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:52.954049 6376 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:52.940141 8205 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:52.940970 6905 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:52.957295 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.934283 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:52.957711 6376 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:58:52.952958 8156 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.060876 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:52.952641 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.061517 6373 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:53.003054 6195 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:52.935954 6307 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:53.040582 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:53.063627 8111 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.041059 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.002516 6273 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:52.961366 6374 raft_consensus.cc:1240] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
W20260620 19:58:53.070510 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 68086.4us
W20260620 19:58:53.066457 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 63477.8us
I20260620 19:58:53.068377 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.073752 6376 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:53.074734 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.094221 6230 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:53.075536 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:52.892248 8132 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:52.890473 8196 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:53.120131 6195 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:53.122462 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.093735 8205 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:53.125962 8240 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260620 19:58:53.095053 8099 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:58:53.128907 8212 consensus_meta.cc:287] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.510s user 0.009s sys 0.002s
I20260620 19:58:53.042467 8147 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.000223 7965 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:53.126873 6273 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:53.127979 8041 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:53.108067 8058 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.398s user 0.010s sys 0.029s
I20260620 19:58:53.076460 8160 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:53.058758 8080 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:58:53.145677 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 412566us
I20260620 19:58:53.121778 8196 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:53.070078 8165 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.064064 8202 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.152351 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.119746 8023 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.152163 8112 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260620 19:58:53.166993 8058 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.449s user 0.013s sys 0.031s
I20260620 19:58:53.169782 8196 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:53.184787 8132 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:53.185133 8245 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260620 19:58:53.186715 8244 log.cc:927] Time spent T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260620 19:58:53.143850 8205 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.164678 8160 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:53.147920 8099 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.151147 8212 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.155478 7965 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.230800 7965 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.231565 6228 raft_consensus.cc:1240] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:58:53.232570 8212 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.181348 8112 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:53.233669 8112 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:53.195015 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.198405 6298 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:52.882081 6379 rpcz_store.cc:276] 0620 19:58:51.639135 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.396789 (+757654us) service_pool.cc:224] Handling call
0620 19:58:52.397167 (+ 378us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:52.397526 (+ 359us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:52.397568 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:52.398069 (+ 501us) log.cc:844] Serialized 26 byte log entry
0620 19:58:52.578494 (+180425us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:58:52.685529 (+107035us) log.cc:844] Serialized 14 byte log entry
0620 19:58:52.732195 (+ 46666us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:52.735150 (+ 2955us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:52.740717 (+ 5567us) raft_consensus.cc:1685] finished
0620 19:58:52.747958 (+ 7241us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:52.761109 (+ 13151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4080,"prepare.queue_time_us":56438,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":12926,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":12273,"thread_start_us":255010,"threads_started":2,"wal-append.queue_time_us":209214}
I20260620 19:58:53.201370 6380 raft_consensus.cc:1240] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:53.202981 8023 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:53.206884 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 355) took 1896 ms. Trace:
I20260620 19:58:53.181483 8058 raft_consensus.cc:697] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:53.208031 8052 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.210711 8205 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.213653 8156 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:53.176859 8080 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.219097 8202 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:53.219945 6376 raft_consensus.cc:1275] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:53.267109 6344 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.196597 8165 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:53.224238 7991 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.225322 8099 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.226572 6372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1777) took 1839 ms. Trace:
I20260620 19:58:53.276722 6372 rpcz_store.cc:276] 0620 19:58:51.387603 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.928362 (+540759us) service_pool.cc:224] Handling call
0620 19:58:51.928707 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:52.019529 (+ 90822us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:52.021946 (+ 2417us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:52.024439 (+ 2493us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:58:52.027090 (+ 2651us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.226297 (+1199207us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b54000249d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:53.226353 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:53.226505 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188827008}
I20260620 19:58:53.260869 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 497) took 1128 ms. Trace:
I20260620 19:58:53.278421 8023 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.278666 8052 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:53.278792 6298 rpcz_store.cc:276] 0620 19:58:52.128631 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.130673 (+ 2042us) service_pool.cc:224] Handling call
0620 19:58:53.197995 (+1067322us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b5800003098. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b579169c 00007fa6b579125b 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:58:53.198189 (+ 194us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:53.256392 (+ 58203us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.256617 (+ 225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28531584}
I20260620 19:58:53.227434 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 726) took 1441 ms. Trace:
W20260620 19:58:53.258138 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 138052us
I20260620 19:58:53.256990 6375 rpcz_store.cc:276] 0620 19:58:51.310434 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.810911 (+500477us) service_pool.cc:224] Handling call
0620 19:58:52.194461 (+383550us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:52.276434 (+ 81973us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.206618 (+930184us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54003c2b50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:53.206801 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":881224960}
I20260620 19:58:53.190389 8160 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:53.259409 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.269353 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.271240 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.258208 7965 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.261564 8176 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:53.273667 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.274658 6344 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.280550 6910 raft_consensus.cc:1240] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:53.224341 7995 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.293207 6344 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.295943 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:53.296653 6225 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:53.257367 8125 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.298105 6230 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:53.298688 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:53.305054 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.286728 8144 raft_consensus.cc:3055] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:53.302874 7995 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:53.268321 8196 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.310334 6228 raft_consensus.cc:1275] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:53.313525 8196 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:53.313838 8189 log.cc:927] Time spent T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 1.482s user 0.004s sys 0.000s
I20260620 19:58:53.316617 8091 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.317782 6381 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:53.319523 8196 raft_consensus.cc:2955] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:53.321729 8080 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.322149 8058 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.286065 6903 rpcz_store.cc:276] 0620 19:58:51.785685 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.016333 (+230648us) service_pool.cc:224] Handling call
0620 19:58:52.039079 (+ 22746us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:52.408644 (+369565us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:52.410923 (+ 2279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:52.419579 (+ 8656us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:52.421061 (+ 1482us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.227172 (+806111us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b5400531454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:53.227227 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:53.227363 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":808254848}
I20260620 19:58:53.303251 8238 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:58:53.297354 8176 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:53.328336 8239 log.cc:927] Time spent T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.391s user 0.001s sys 0.000s
I20260620 19:58:53.327153 6225 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:53.296998 8001 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260620 19:58:53.322892 8146 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.294844 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.307451 8197 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.329118 8112 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:53.343602 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 45041.5us
W20260620 19:58:53.340904 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 77708.4us
I20260620 19:58:53.345566 8125 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.346454 6229 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:53.347182 6307 raft_consensus.cc:1275] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:53.306810 8224 consensus_queue.cc:1059] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260620 19:58:53.348982 8224 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:58:53.349371 8224 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:53.350256 8224 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.351717 8224 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:53.309984 8144 raft_consensus.cc:740] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:53.347334 6379 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:53.354378 8144 consensus_queue.cc:260] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.355619 7965 consensus_queue.cc:1048] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 19:58:53.355907 8144 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:53.357836 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:53.335556 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 75100us
I20260620 19:58:53.359290 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.359848 6375 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:53.360241 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.360656 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:53.361523 6273 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:53.361084 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.363979 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:53.365890 6306 raft_consensus.cc:1275] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:53.366875 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:53.367400 6225 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:53.370893 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:53.371454 6224 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:53.372280 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:53.373078 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:53.373452 6224 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:53.371476 6375 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:53.375891 6273 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:53.377673 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.378939 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.379590 8258 consensus_queue.cc:1059] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:53.380205 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.380853 6298 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:53.378329 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.367570 6230 raft_consensus.cc:2955] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:53.382893 6345 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:53.383370 6225 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:58:53.383934 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:53.384634 6300 raft_consensus.cc:2376] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4. Re-sending same reply.
I20260620 19:58:53.390614 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.392000 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.388640 6344 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:53.341451 8176 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:53.395231 6303 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:53.397305 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.392797 6375 raft_consensus.cc:2368] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:58:53.358227 8165 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:53.402568 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.486433 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:53.403877 6228 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:53.461778 6225 raft_consensus.cc:1240] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:53.487452 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:53.490332 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:53.374740 8205 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:53.491847 6225 raft_consensus.cc:1275] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:53.494733 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:53.495430 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:53.497829 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:53.499146 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.506754 6299 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260620 19:58:53.508249 6299 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:58:53.394955 7965 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:53.513372 6225 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:53.515179 6225 raft_consensus.cc:1240] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:53.517133 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.518185 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.397586 8040 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:53.521874 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 145105us
I20260620 19:58:53.395630 8259 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:53.400791 8220 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:53.439952 8176 raft_consensus.cc:3055] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:53.466315 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:53.538005 6906 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:58:53.538455 6906 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 694) took 1763 ms (client timeout 2035 ms). Trace:
W20260620 19:58:53.539196 6906 rpcz_store.cc:269] 0620 19:58:51.775339 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.821268 (+ 45929us) service_pool.cc:224] Handling call
0620 19:58:53.459675 (+1638407us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:53.538390 (+ 78715us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14366976}
I20260620 19:58:53.534560 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.541074 6375 raft_consensus.cc:2393] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:53.541947 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.438862 8238 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:53.543550 8238 raft_consensus.cc:697] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:53.365548 8156 consensus_queue.cc:1059] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.140s
W20260620 19:58:53.509397 8206 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.007s sys 0.014s
W20260620 19:58:53.550637 8206 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.007s sys 0.015s
I20260620 19:58:53.552673 8206 raft_consensus.cc:3055] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:53.553128 8206 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:53.554072 8206 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.555572 8206 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:53.560983 8176 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:53.563090 8176 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.567118 6375 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 8: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:53.568677 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.572868 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.548815 8238 consensus_queue.cc:237] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.592387 8040 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.594475 6306 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:53.592752 8091 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.500540 8205 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:53.402906 8080 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.137s
I20260620 19:58:53.599051 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:53.495551 8001 consensus_queue.cc:1048] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.101s
I20260620 19:58:53.398885 8146 consensus_queue.cc:1059] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.136s
W20260620 19:58:53.606772 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 112073us
I20260620 19:58:53.523088 6373 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:53.589409 8176 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:53.618117 6198 leader_election.cc:400] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:58:53.618559 6198 leader_election.cc:403] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260620 19:58:53.496483 8157 consensus_queue.cc:1059] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.102s
I20260620 19:58:53.600149 8205 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:53.621755 8040 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:53.635382 6198 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:53.606325 8148 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:53.620805 8240 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:53.595939 8259 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.598559 8257 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260620 19:58:53.638654 8112 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:53.606117 8160 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.644407 8240 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:53.651819 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.665481 8112 raft_consensus.cc:3055] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:53.647228 8040 consensus_queue.cc:1059] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.020s
I20260620 19:58:53.670940 8040 raft_consensus.cc:3055] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:53.671273 8040 raft_consensus.cc:740] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:53.672030 6375 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:53.670367 8112 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:53.644860 8148 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.627594 8165 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.681603 8160 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.682770 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1786) took 1575 ms. Trace:
I20260620 19:58:53.683219 6380 rpcz_store.cc:276] 0620 19:58:52.107835 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.690671 (+582836us) service_pool.cc:224] Handling call
0620 19:58:52.731482 (+ 40811us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:53.256314 (+524832us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.682500 (+426186us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b5400045bd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:53.682701 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1053733248}
I20260620 19:58:53.684059 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:53.676250 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1781) took 2283 ms. Trace:
I20260620 19:58:53.663100 8240 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:53.649140 8157 consensus_queue.cc:1048] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260620 19:58:53.678721 8268 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:53.679023 8260 log.cc:927] Time spent T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260620 19:58:53.678613 8040 consensus_queue.cc:260] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.686053 6377 rpcz_store.cc:276] 0620 19:58:51.392633 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.301852 (+909219us) service_pool.cc:224] Handling call
0620 19:58:52.302221 (+ 369us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:52.374385 (+ 72164us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:52.374413 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:52.374453 (+ 40us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:58:52.374482 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.676010 (+1301528us) spinlock_profiling.cc:243] Waited 71.9 ms on lock 0x7b54001225d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:53.676053 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:53.676177 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":165480832}
I20260620 19:58:53.693540 6377 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260620 19:58:53.695819 6377 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:53.671815 8222 raft_consensus.cc:3055] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:53.699105 8112 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.702204 8268 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.710999 8040 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:53.712200 8259 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.712769 8259 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:58:53.703243 8222 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:53.715022 6223 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260620 19:58:53.716931 6229 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1791) took 1935 ms (client timeout 2087 ms). Trace:
W20260620 19:58:53.717442 6229 rpcz_store.cc:269] 0620 19:58:51.781952 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.785280 (+ 3328us) service_pool.cc:224] Handling call
0620 19:58:53.714528 (+1929248us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540004b5d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:53.716855 (+ 2327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3585464192}
I20260620 19:58:53.681633 8148 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:53.715390 8222 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.723263 8222 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:53.724133 6231 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:58:53.728576 6302 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1390) took 2066 ms (client timeout 1558 ms). Trace:
W20260620 19:58:53.729075 6302 rpcz_store.cc:269] 0620 19:58:51.662119 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.662306 (+ 187us) service_pool.cc:224] Handling call
0620 19:58:53.728353 (+2066047us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b54003a6e54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:53.728488 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177270656}
W20260620 19:58:53.740278 8184 log.cc:927] Time spent T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 1.921s user 0.003s sys 0.001s
I20260620 19:58:53.745074 6231 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:53.748418 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.749080 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.749761 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.750999 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:53.751266 6381 raft_consensus.cc:1240] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:58:53.750402 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.753436 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:58:53.753747 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 231154us
I20260620 19:58:53.757530 6877 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:53.755522 6345 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:53.777770 6229 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:53.772883 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:53.785411 6345 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:53.735008 8112 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:53.726511 8148 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.787489 6226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1894) took 1917 ms. Trace:
I20260620 19:58:53.790068 6228 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:53.783461 8091 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:53.793089 6226 rpcz_store.cc:276] 0620 19:58:51.870079 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.870519 (+ 440us) service_pool.cc:224] Handling call
0620 19:58:53.786761 (+1916242us) spinlock_profiling.cc:243] Waited 881 us on lock 0x7b5800000c98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57916ee 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:58:53.786905 (+ 144us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:53.787091 (+ 186us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:53.787127 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:53.787181 (+ 54us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:58:53.787226 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.787298 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:53.787420 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2038400}
I20260620 19:58:53.771552 8224 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.771162 8080 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:58:53.782346 8202 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:53.782501 8258 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:53.787582 7925 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:53.803787 8111 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260620 19:58:53.788563 8176 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.790304 8144 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:53.795287 8091 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.802364 8001 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:58:53.801184 8146 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260620 19:58:53.812150 7925 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:53.831252 8144 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:53.795819 8249 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260620 19:58:53.803527 8205 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.850896 6875 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.856534 6875 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.833282 8207 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:53.812383 8202 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.841349 8001 raft_consensus.cc:697] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:53.844887 7925 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:53.870188 6875 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.854805 8144 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:53.840010 8176 consensus_queue.cc:1059] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.024s
I20260620 19:58:53.814544 8258 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:53.879942 6875 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:53.873876 8207 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260620 19:58:53.945125 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:53.922897 8123 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:53.971745 8208 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.246s user 0.001s sys 0.001s
I20260620 19:58:53.877702 8156 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260620 19:58:53.979805 8207 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.988858 8207 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:58:53.986637 8001 consensus_queue.cc:237] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.989913 8249 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:53.992525 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 689) took 2231 ms. Trace:
I20260620 19:58:53.993026 6909 rpcz_store.cc:276] 0620 19:58:51.761321 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.762088 (+ 767us) service_pool.cc:224] Handling call
0620 19:58:51.762350 (+ 262us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:51.763042 (+ 692us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:51.763066 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:51.768247 (+ 5181us) log.cc:844] Serialized 26 byte log entry
0620 19:58:52.039091 (+270844us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:58:52.042368 (+ 3277us) log.cc:844] Serialized 14 byte log entry
0620 19:58:52.103495 (+ 61127us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:52.103565 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:53.704508 (+1600943us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b18002faca0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:58:53.992108 (+287600us) raft_consensus.cc:1685] finished
0620 19:58:53.992252 (+ 144us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:53.992421 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60888,"prepare.queue_time_us":3151,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3012,"raft.queue_time_us":141581,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":9745,"spinlock_wait_cycles":29603072,"thread_start_us":239175,"threads_started":1,"wal-append.queue_time_us":329032}
I20260620 19:58:53.980448 8123 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:53.978680 8268 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:54.000309 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 391057us
I20260620 19:58:53.997254 8202 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:54.027432 7933 consensus_queue.cc:1059] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260620 19:58:54.038228 8215 raft_consensus.cc:3055] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:54.044785 8215 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:58:54.040599 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.045373 8240 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.045570 8215 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.046912 8258 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:54.010496 8091 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:54.050455 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.058555 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.059674 6910 raft_consensus.cc:2410] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:58:54.062850 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.011691 8137 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
W20260620 19:58:54.037689 8132 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.007s sys 0.027s
I20260620 19:58:54.047711 8205 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.039969 8123 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:54.068040 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.068347 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.070614 6910 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:54.073604 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.063799 8215 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:54.038488 8196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:46180 (request call id 13) took 7610 ms. Trace:
W20260620 19:58:54.067142 8132 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.008s sys 0.027s
I20260620 19:58:54.119061 8240 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:54.138630 8292 log.cc:927] Time spent T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260620 19:58:54.140419 8249 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:54.141172 8249 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.114295 8196 rpcz_store.cc:276] 0620 19:58:46.401042 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:46.401940 (+ 898us) service_pool.cc:224] Handling call
0620 19:58:46.441169 (+ 39229us) log.cc:844] Serialized 506 byte log entry
0620 19:58:46.638594 (+197425us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b4c00086dc0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b6b84120 00007fa6b6b95129 00007fa6b6b9583f 00007fa6b6b599ab 00007fa6b6b8763c 00007fa6b6b87afe 00007fa6b6e1566b 00007fa6b8816761 00007fa6b881464b 00007fa6b881b133 00007fa6b881b0b9
0620 19:58:46.651755 (+ 13161us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b180032d8e0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b574340a 00007fa6bee885d9 00007fa6b57cb240 00007fa6b57c5ce4 00007fa6b57c5c55 00007fa6b8813eeb 00007fa6b8898f43 00007fa6b88c0b32 00007fa6bd881429
0620 19:58:46.663440 (+ 11685us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b5400114020. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b8833522 00007fa6b881428a 00007fa6b881b133 00007fa6b881b0b9 00007fa6b881b049 00007fa6b881b011 00007fa6b881a1dd 0000000000389354 0000000000389199 00007fa6b57c8c75
0620 19:58:54.010551 (+7347111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17182,"raft.queue_time_us":107255,"raft.run_cpu_time_us":4306,"raft.run_wall_time_us":104790,"spinlock_wait_cycles":34062848,"thread_start_us":91395,"threads_started":1,"wal-append.queue_time_us":125753,"wal-append.run_cpu_time_us":3158,"wal-append.run_wall_time_us":1210837}
I20260620 19:58:54.152343 8249 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:54.153244 8249 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:54.156759 8144 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.159667 8144 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.154196 8249 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.162017 8249 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:54.190958 6196 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.006s sys 0.013s
W20260620 19:58:54.191920 6196 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.006s sys 0.013s
W20260620 19:58:54.194500 6198 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.004s sys 0.020s
W20260620 19:58:54.194490 6197 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.023s sys 0.036s
W20260620 19:58:54.195216 6198 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s user 0.004s sys 0.020s
W20260620 19:58:54.195365 6197 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.023s sys 0.036s
W20260620 19:58:54.198173 6224 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.013s sys 0.016s
I20260620 19:58:54.045534 6373 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
W20260620 19:58:54.201900 6224 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.014s sys 0.016s
W20260620 19:58:54.202862 6380 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.008s sys 0.018s
W20260620 19:58:54.203367 6380 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.008s sys 0.019s
W20260620 19:58:54.206640 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:54.212029 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 331856us
I20260620 19:58:54.075304 7965 raft_consensus.cc:697] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:58:54.040987 6305 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1402) took 1906 ms (client timeout 1691 ms). Trace:
W20260620 19:58:54.217118 6305 rpcz_store.cc:269] 0620 19:58:52.134965 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.246082 (+111117us) service_pool.cc:224] Handling call
0620 19:58:54.040799 (+1794717us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b5400332a54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:54.040921 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":758129408}
W20260620 19:58:54.197315 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 561714us
I20260620 19:58:54.220882 6198 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.222364 6875 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:54.217308 7965 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:54.244805 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 230373us
W20260620 19:58:54.251885 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.252771 6344 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:53.222281 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:58:54.259214 8039 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:54.260313 8039 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:54.262763 6196 leader_election.cc:336] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 1.778s (SENT)
I20260620 19:58:54.263985 8132 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:54.263408 6126 catalog_manager.cc:5696] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:54.264837 8132 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.266738 8290 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:54.271138 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1912) took 1311 ms. Trace:
I20260620 19:58:54.274443 6231 rpcz_store.cc:276] 0620 19:58:52.959289 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.959427 (+ 138us) service_pool.cc:224] Handling call
0620 19:58:54.270680 (+1311253us) spinlock_profiling.cc:243] Waited 657 ms on lock 0x7b5400285d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:54.270877 (+ 197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1510956800}
I20260620 19:58:54.255453 7933 raft_consensus.cc:3055] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
W20260620 19:58:54.287664 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 502771us
I20260620 19:58:54.291193 8286 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:54.288975 7933 raft_consensus.cc:740] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:54.295624 6910 raft_consensus.cc:1240] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:54.295745 7933 consensus_queue.cc:260] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:54.301625 6130 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45406 (request call id 266) took 6673 ms. Trace:
I20260620 19:58:54.286691 8290 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:54.303545 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 51265us
W20260620 19:58:54.303248 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 88940.2us
I20260620 19:58:54.310364 8258 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:54.315469 6910 raft_consensus.cc:1240] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:54.315069 6126 catalog_manager.cc:5696] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 2 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:54.316927 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:54.303437 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 359069us
I20260620 19:58:54.307313 7933 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:54.304350 6877 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:58:54.302589 8112 consensus_meta.cc:287] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.506s user 0.013s sys 0.001s
W20260620 19:58:54.328929 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 108261us
I20260620 19:58:54.309101 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.306874 8290 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.304669 8039 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.339955 8133 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:54.340034 8215 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:54.341727 6877 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:54.320749 6126 catalog_manager.cc:5696] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 4 to 5, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 5 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:54.334015 8258 raft_consensus.cc:697] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:54.346905 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.348701 8258 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.349344 6198 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.357095 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.362705 8290 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:54.363541 8290 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.302060 6130 rpcz_store.cc:276] 0620 19:58:47.627944 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:47.629954 (+ 2010us) service_pool.cc:224] Handling call
0620 19:58:54.301286 (+6671332us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:58:50.879584 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:58:50.880390 (+ 806us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:58:50.880434 (+ 44us) write_op.cc:435] Acquired schema lock
0620 19:58:50.880460 (+ 26us) tablet.cc:662] Decoding operations
0620 19:58:50.881409 (+ 949us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:58:50.881507 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0620 19:58:50.881546 (+ 39us) tablet.cc:685] Acquiring locks for 6 operations
0620 19:58:50.882190 (+ 644us) tablet.cc:701] Row locks acquired
0620 19:58:50.882215 (+ 25us) write_op.cc:260] PREPARE: finished
0620 19:58:50.882393 (+ 178us) write_op.cc:270] Start()
0620 19:58:50.882528 (+ 135us) write_op.cc:276] Timestamp: P: 1781985530882345 usec, L: 0
0620 19:58:50.882554 (+ 26us) op_driver.cc:348] REPLICATION: starting
0620 19:58:50.899780 (+ 17226us) log.cc:844] Serialized 2452 byte log entry
0620 19:58:51.783338 (+883558us) op_driver.cc:464] REPLICATION: finished
0620 19:58:52.015678 (+232340us) write_op.cc:301] APPLY: starting
0620 19:58:52.021792 (+ 6114us) tablet.cc:1367] starting BulkCheckPresence
0620 19:58:52.041308 (+ 19516us) tablet.cc:1370] finished BulkCheckPresence
0620 19:58:52.121464 (+ 80156us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:58:52.357398 (+235934us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:58:52.357597 (+ 199us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0620 19:58:52.357650 (+ 53us) write_op.cc:312] APPLY: finished
0620 19:58:52.357892 (+ 242us) log.cc:844] Serialized 53 byte log entry
0620 19:58:52.358062 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:58:52.358172 (+ 110us) write_op.cc:454] Released schema lock
0620 19:58:52.358211 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0620 19:58:52.358550 (+ 339us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:58:52.358581 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16146,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":210914,"apply.run_cpu_time_us":4746,"apply.run_wall_time_us":348394,"num_ops":6,"prepare.queue_time_us":103782,"prepare.run_cpu_time_us":32467,"prepare.run_wall_time_us":277124,"raft.queue_time_us":391001,"raft.run_cpu_time_us":48315,"raft.run_wall_time_us":210200,"replication_time_us":879332,"thread_start_us":765199,"threads_started":4,"wal-append.queue_time_us":405755,"wal-append.run_cpu_time_us":25169,"wal-append.run_wall_time_us":2056115}]]}
I20260620 19:58:54.379514 6910 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:58:54.367303 8132 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.388156 6909 raft_consensus.cc:2368] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:54.388782 8133 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:54.398949 8299 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:54.401777 8290 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:54.402433 7925 consensus_meta.cc:287] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.524s user 0.003s sys 0.016s
I20260620 19:58:54.403119 7925 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:54.406013 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 56858.7us
W20260620 19:58:54.407081 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 75878.7us
I20260620 19:58:54.408049 6344 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:54.409444 8298 log.cc:927] Time spent T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20260620 19:58:54.417604 6911 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 685) took 2665 ms (client timeout 1755 ms). Trace:
I20260620 19:58:54.419919 8215 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:54.420207 6877 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.420699 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:54.420905 8215 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.422676 8301 consensus_queue.cc:1059] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260620 19:58:54.436846 8299 raft_consensus.cc:3055] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:54.424135 8242 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:54.437292 8299 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:58:54.438016 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.438913 8299 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:54.440745 8305 log.cc:927] Time spent T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260620 19:58:54.440557 8299 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:54.426941 8193 consensus_queue.cc:1059] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260620 19:58:54.427374 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:54.444231 8224 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:54.444979 8224 raft_consensus.cc:697] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:58:54.429947 6911 rpcz_store.cc:269] 0620 19:58:51.752224 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.752321 (+ 97us) service_pool.cc:224] Handling call
0620 19:58:54.417045 (+2664724us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54005eef54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:54.417346 (+ 301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3882057472}
I20260620 19:58:54.431298 8215 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.431497 8058 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:54.448767 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1675) took 1069 ms. Trace:
I20260620 19:58:54.449194 6306 rpcz_store.cc:276] 0620 19:58:53.378993 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:53.397113 (+ 18120us) service_pool.cc:224] Handling call
0620 19:58:54.448326 (+1051213us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54001167d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:54.448496 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":451892992}
I20260620 19:58:54.450157 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1901) took 1820 ms. Trace:
I20260620 19:58:54.422676 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.446547 8224 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.433171 8207 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:58:54.454365 6908 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:54.429876 8132 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260620 19:58:54.432367 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:54.433715 8001 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:58:54.433110 6344 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:54.449088 8176 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:54.422542 6877 leader_election.cc:304] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.450623 6227 rpcz_store.cc:276] 0620 19:58:52.629489 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.629623 (+ 134us) service_pool.cc:224] Handling call
0620 19:58:52.634780 (+ 5157us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:52.736072 (+101292us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:54.449626 (+1713554us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54003723d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:54.449942 (+ 316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":231355264}
I20260620 19:58:54.456902 8207 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:58:54.467945 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 45577.5us
I20260620 19:58:54.468333 8207 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.469769 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.470532 8207 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:58:54.471235 8191 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 19:58:54.474524 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.475591 6907 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:54.476018 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.476874 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.470883 8058 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:54.479300 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.479887 6911 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:54.482355 6910 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:54.487061 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.488283 6911 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:54.434850 7925 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.492518 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 508) took 2086 ms. Trace:
I20260620 19:58:54.495260 6304 rpcz_store.cc:276] 0620 19:58:52.371006 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.372909 (+ 1903us) service_pool.cc:224] Handling call
0620 19:58:53.738938 (+1366029us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:53.741061 (+ 2123us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:53.741095 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:53.741151 (+ 56us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:53.741191 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:53.741256 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:54.456677 (+715421us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54003a6e50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:54.456896 (+ 219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3038265728}
W20260620 19:58:54.496469 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 63473.9us
I20260620 19:58:54.498857 8001 raft_consensus.cc:2764] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260620 19:58:54.506011 8125 consensus_queue.cc:1059] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:54.506904 7991 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:54.507252 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.507627 8125 raft_consensus.cc:3055] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:54.508021 8125 raft_consensus.cc:740] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:54.509161 8125 consensus_queue.cc:260] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.506323 6271 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.511446 8125 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:54.511642 8157 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260620 19:58:54.521520 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.523577 6223 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1915) took 1228 ms. Trace:
I20260620 19:58:54.524005 6223 rpcz_store.cc:276] 0620 19:58:53.295220 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:53.296875 (+ 1655us) service_pool.cc:224] Handling call
0620 19:58:53.714686 (+417811us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:53.715405 (+ 719us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:54.523222 (+807817us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b540004b5d0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:54.523420 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":949160320}
I20260620 19:58:54.524935 6126 catalog_manager.cc:5696] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 3, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:54.531144 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.532861 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1921) took 1126 ms. Trace:
I20260620 19:58:54.533138 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.533869 6307 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:54.533269 6230 rpcz_store.cc:276] 0620 19:58:53.366657 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:53.367008 (+ 351us) service_pool.cc:224] Handling call
0620 19:58:53.367265 (+ 257us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:53.367465 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 4
0620 19:58:53.455153 (+ 87688us) log.cc:844] Serialized 14 byte log entry
0620 19:58:54.447146 (+991993us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b18003f2bc0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b846185c 00007fa6b57abc8b 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c5c55 00007fa6bc2b4247 00007fa6b88c4d87 00007fa6b88d449f
0620 19:58:54.447395 (+ 249us) log.cc:844] Serialized 14 byte log entry
0620 19:58:54.450493 (+ 3098us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:54.450583 (+ 90us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:58:54.450736 (+ 153us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:54.459025 (+ 8289us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:54.493235 (+ 34210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52954,"prepare.queue_time_us":90976,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":213483,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":9967,"spinlock_wait_cycles":12848384,"thread_start_us":128683,"threads_started":2}
I20260620 19:58:54.525669 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:54.542068 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 43509us
I20260620 19:58:54.543596 6911 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:54.546846 6908 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:54.434108 8290 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.107s
I20260620 19:58:54.555528 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.561475 6910 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:54.565260 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:54.580158 6378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 1789) took 2428 ms (client timeout 2088 ms). Trace:
W20260620 19:58:54.580659 6378 rpcz_store.cc:269] 0620 19:58:52.120866 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:52.270337 (+149471us) service_pool.cc:224] Handling call
0620 19:58:54.493390 (+2223053us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b54000456d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:54.549002 (+ 55612us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1661513600}
I20260620 19:58:54.583735 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:54.588536 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:54.591231 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.594337 6910 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:54.598948 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.600214 6230 raft_consensus.cc:2393] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:54.601279 6907 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260620 19:58:54.608584 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 56608.2us
I20260620 19:58:54.611763 6911 raft_consensus.cc:1240] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:54.607800 6306 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:54.616179 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.617020 8165 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:54.617122 6902 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:58:54.617988 8165 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.620450 8165 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.623018 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1801) took 1533 ms. Trace:
I20260620 19:58:54.623190 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:54.623397 8273 consensus_queue.cc:1059] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:54.623467 6374 rpcz_store.cc:276] 0620 19:58:53.072527 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:53.075227 (+ 2700us) service_pool.cc:224] Handling call
0620 19:58:53.075472 (+ 245us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:53.121300 (+ 45828us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:53.121346 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:53.121401 (+ 55us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:58:53.121448 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:54.587384 (+1465936us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b54000244d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:54.589672 (+ 2288us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:54.605906 (+ 16234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":104896896}
I20260620 19:58:54.629307 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.631420 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:54.638355 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.638643 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.639506 6224 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:54.641172 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.646806 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.653017 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.643087 8165 raft_consensus.cc:3055] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:54.665202 6910 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:54.664402 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.667572 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:54.667148 6907 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:58:54.669526 6345 leader_election.cc:336] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 1.755s (SENT)
I20260620 19:58:54.669472 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:54.672264 6908 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:54.674872 6345 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:58:54.675402 8310 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260620 19:58:54.665310 8165 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:54.675753 6196 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.683676 6224 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:54.691380 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.671193 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.693459 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.695744 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.646176 8132 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260620 19:58:54.695333 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.699364 6905 raft_consensus.cc:2376] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3. Re-sending same reply.
I20260620 19:58:54.701264 6910 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:54.702730 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:54.703536 6373 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:54.695423 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:54.706787 6378 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:58:54.707715 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 159304us
I20260620 19:58:54.712680 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.714969 8144 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:54.715230 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.716488 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.717238 6908 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:54.717429 8144 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260620 19:58:54.719141 8306 log.cc:927] Time spent T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.338s user 0.005s sys 0.000s
I20260620 19:58:54.719233 6301 raft_consensus.cc:1240] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:54.721104 6299 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
W20260620 19:58:54.722970 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 80432.5us
I20260620 19:58:54.723788 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:54.724218 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.725101 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.725023 8251 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:54.727026 6302 raft_consensus.cc:1240] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:54.727129 8251 consensus_queue.cc:1059] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:54.727519 7933 consensus_queue.cc:1059] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:54.728725 8039 raft_consensus.cc:3055] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:54.729146 8039 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:54.729228 6225 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:54.731020 7933 raft_consensus.cc:3055] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:54.731426 7933 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:54.731195 8165 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:54.732306 8165 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:58:54.732616 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 61051.6us
I20260620 19:58:54.736104 6909 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:54.678436 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.736958 6908 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:54.726357 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.738207 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.740913 6196 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:54.732576 8259 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:58:54.742388 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:54.743687 6911 raft_consensus.cc:2368] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:54.722584 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.714506 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.746516 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.747361 6908 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:54.749222 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.770970 6229 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:58:54.771857 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.772002 6909 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:54.772934 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:54.773729 6909 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:58:54.749897 8224 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:54.749168 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:54.737109 8001 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:54.753986 6373 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:54.754492 6196 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:54.784032 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.788715 8224 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:54.792100 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 46711.4us
I20260620 19:58:54.756449 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.756604 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:54.793128 6380 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:54.793774 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.757362 8146 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:54.732429 7933 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.795717 7933 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:54.796159 6196 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:54.792243 8290 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:54.798354 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.798868 8290 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:54.799329 8290 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
W20260620 19:58:54.799995 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.801499 8001 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.804409 8001 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.754859 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:54.806656 6226 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:54.773981 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.829370 6229 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:54.831409 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.834208 6196 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:54.835408 8070 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:54.835705 6196 leader_election.cc:400] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:54.835824 8070 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:54.836717 8070 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:54.836742 8224 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.842236 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.745535 8259 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:54.846776 8259 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:58:54.852088 6229 raft_consensus.cc:2393] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:54.852945 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:54.853529 6229 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:54.857440 8133 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:54.836165 6196 leader_election.cc:403] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260620 19:58:54.856997 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:54.859933 6304 raft_consensus.cc:1275] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:54.861562 6304 raft_consensus.cc:1275] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:54.865795 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:54.747853 6906 raft_consensus.cc:2376] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3. Re-sending same reply.
W20260620 19:58:54.872570 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.877596 6224 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:54.878317 6911 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:54.884799 6226 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
W20260620 19:58:54.894002 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 229468us
I20260620 19:58:54.896690 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.876663 8133 raft_consensus.cc:697] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:54.900336 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.894886 6298 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:58:54.896281 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:54.844131 8290 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.914039 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.899659 8132 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3.
I20260620 19:58:54.913492 6226 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4. Ops: []
W20260620 19:58:54.919757 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:54.930013 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:54.912014 7991 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:54.926777 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:54.919078 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:54.939772 6232 raft_consensus.cc:2393] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 2: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:54.940140 6379 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:54.940178 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:54.941010 6378 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:54.941797 8125 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:54.942430 8125 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.943809 6345 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:54.944721 8125 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:54.945516 8215 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:54.945956 8215 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:54.946347 8215 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:54.946399 6375 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:54.906133 8224 consensus_queue.cc:1059] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260620 19:58:54.953841 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:54.954578 6381 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:54.929294 8133 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.922909 8247 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260620 19:58:54.951272 6226 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:54.959300 6906 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:54.919013 8273 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:54.963163 8307 log.cc:927] Time spent T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.347s user 0.002s sys 0.000s
I20260620 19:58:54.963047 8273 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:54.969048 8207 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:54.969691 8207 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.970887 8267 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:54.915347 8001 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260620 19:58:54.971513 8267 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.973694 8070 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:54.974033 8267 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:54.978948 6225 raft_consensus.cc:1240] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:54.979992 6874 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:54.981060 8207 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:54.981763 8258 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:58:54.982339 8176 consensus_queue.cc:1059] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:54.982856 8156 consensus_queue.cc:1059] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:54.982957 8258 consensus_queue.cc:1059] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:54.983883 8058 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:54.983305 8176 raft_consensus.cc:3055] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:54.984426 8058 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:54.984632 8176 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:54.984894 6345 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:54.983922 6377 raft_consensus.cc:1275] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:58:54.983919 8258 consensus_queue.cc:1243] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260620 19:58:54.971596 6195 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:54.959417 6905 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:54.990058 8132 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:54.990495 8132 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:54.990940 8132 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:54.994336 6226 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:58:54.996435 6377 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260620 19:58:54.997835 6273 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:54.938743 8146 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260620 19:58:55.004796 6908 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:55.005632 8146 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:55.006037 8146 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:55.006457 8146 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:55.009191 6228 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260620 19:58:55.009588 6228 pending_rounds.cc:85] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 4
I20260620 19:58:55.009887 6228 pending_rounds.cc:107] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260620 19:58:55.004180 8070 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.010176 6228 raft_consensus.cc:2889] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:58:54.978950 6380 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:54.985554 8058 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.038326 8058 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:55.038933 6198 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:55.039330 8290 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.040258 8290 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:58:55.040367 6198 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:55.040902 8290 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:55.041433 6198 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:54.985709 8176 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.031659 6379 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:55.045053 6198 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:55.046167 6198 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:55.046448 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 53278.8us
I20260620 19:58:55.048552 8112 consensus_queue.cc:1059] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260620 19:58:55.049785 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.055527 6195 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:55.030635 8165 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.042119 8290 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.089144 8001 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:58:55.067337 8273 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:55.103169 8273 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:55.103624 8273 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:55.104035 8273 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:55.058118 8070 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:55.115023 8070 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:55.115520 8070 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:58:55.120273 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.074805 8125 raft_consensus.cc:3055] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:55.143661 8125 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:55.144583 8125 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.146049 8125 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:55.140795 8039 consensus_queue.cc:260] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.077060 8242 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:55.078234 8165 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.152141 6229 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:55.078588 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:55.159235 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 323620us
I20260620 19:58:55.167433 6196 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:55.168849 8255 raft_consensus.cc:2764] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260620 19:58:55.153406 8247 raft_consensus.cc:3055] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:58:55.158555 8242 raft_consensus.cc:697] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:55.168972 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:55.169217 6905 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:55.174571 8247 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:55.180261 8039 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:55.181165 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:55.203572 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.204334 8328 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:55.204838 8327 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:55.209926 8328 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:55.185282 8146 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.182766 8259 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:55.233420 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.240285 8242 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.212212 6906 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:55.212363 8327 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:55.243137 8328 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:55.226643 8132 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:55.254835 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 208645us
I20260620 19:58:55.272318 6198 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:55.246050 8165 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.228214 6304 raft_consensus.cc:2955] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:58:55.246759 8327 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:55.270273 6223 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
W20260620 19:58:55.320137 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 240180us
I20260620 19:58:55.315291 7991 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:58:55.333266 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 348294us
I20260620 19:58:55.336580 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:55.337728 8224 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:55.338276 8224 raft_consensus.cc:697] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:55.332190 7991 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:55.308637 8247 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:55.345412 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 295841us
I20260620 19:58:55.344954 8165 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:55.339186 8224 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:55.369315 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 201902us
I20260620 19:58:55.370905 6196 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:55.372915 8247 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:55.388296 6273 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:55.389474 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 117227us
W20260620 19:58:55.385135 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 60034.4us
I20260620 19:58:55.379560 8146 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.380367 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:55.376479 8338 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:58:55.382320 8337 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:55.385349 8259 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:55.404785 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 43873.7us
I20260620 19:58:55.410364 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:55.405483 8112 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:55.411969 6273 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:55.413663 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:55.413908 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.420269 6228 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:55.421682 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:55.421856 6345 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:55.422528 6230 raft_consensus.cc:2410] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:58:55.424868 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:55.433815 8215 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:55.435165 6231 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:58:55.406190 8132 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:55.409261 7925 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:55.439102 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:55.436079 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:55.440482 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.443356 6226 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:55.420656 8112 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
W20260620 19:58:55.450484 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.452021 6381 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:55.449659 6875 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:55.443562 8338 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
W20260620 19:58:55.460924 8329 log.cc:927] Time spent T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260620 19:58:55.461061 7925 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:55.463129 8333 log.cc:927] Time spent T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260620 19:58:55.463447 7925 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:55.460927 8340 consensus_queue.cc:1059] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260620 19:58:55.457682 8342 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:55.464877 8345 raft_consensus.cc:3055] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:55.465227 8342 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:55.466120 8331 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20260620 19:58:55.468142 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:55.455626 8337 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:55.465240 8345 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:55.436363 8001 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260620 19:58:55.474587 6345 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:55.469756 8345 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.455174 8334 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260620 19:58:55.494556 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 50111.5us
I20260620 19:58:55.452417 8346 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:55.506498 8176 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:55.507143 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:55.507730 8215 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:55.508323 8052 raft_consensus.cc:3055] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:55.477494 8267 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 19:58:55.528297 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 53216.5us
I20260620 19:58:55.485608 8334 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:55.538833 8334 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:55.484609 8338 raft_consensus.cc:697] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:55.502717 7925 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:58:55.540104 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 100543us
I20260620 19:58:55.532013 6273 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:55.540676 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 85827.4us
I20260620 19:58:55.521903 8052 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:55.526348 8112 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:55.529066 8165 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:55.547382 8001 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.535406 8340 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.520104 8353 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260620 19:58:55.548426 8345 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:55.547427 8215 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.561147 7991 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.562052 8337 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:55.563346 8347 log.cc:927] Time spent T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20260620 19:58:55.563838 6345 consensus_peers.cc:597] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 -> Peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Couldn't send request to peer 20bd146dc0cb4917a1a5ce05ec609ce1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42a982b7017a41e1b0ca419a82cacf59. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:58:55.555063 8001 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.565824 8338 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.571182 6906 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:55.572500 8125 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:55.576741 6875 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:55.578562 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 75132.9us
W20260620 19:58:55.592213 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 60497.9us
W20260620 19:58:55.593081 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 47753.1us
I20260620 19:58:55.575266 8052 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.598279 6910 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
W20260620 19:58:55.613673 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 218991us
I20260620 19:58:55.609097 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:55.613539 8267 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:58:55.591907 8215 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:55.622620 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 65382.9us
I20260620 19:58:55.591804 8340 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.629480 6273 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:55.598878 8070 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:55.615478 8125 raft_consensus.cc:697] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:55.634465 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:55.623137 8052 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:55.636052 6875 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:55.632513 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:55.641628 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:55.642903 6273 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:55.644939 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:55.645819 6372 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 8: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:55.616727 8359 consensus_queue.cc:1059] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260620 19:58:55.633811 8355 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.639336 6902 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:55.665792 8215 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.622352 8352 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260620 19:58:55.666249 8125 consensus_queue.cc:237] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:55.687216 8332 log.cc:927] Time spent T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.333s user 0.002s sys 0.000s
W20260620 19:58:55.667151 8350 log.cc:927] Time spent T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
W20260620 19:58:55.667758 8330 log.cc:927] Time spent T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.322s user 0.003s sys 0.000s
I20260620 19:58:55.671422 8335 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:55.671814 7991 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.672189 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:55.640380 8058 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:58:55.672870 8337 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:58:55.750469 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 107573us
I20260620 19:58:55.644959 8079 consensus_queue.cc:1059] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260620 19:58:55.673646 8366 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:55.668390 8355 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.639974 8112 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.674825 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 534) took 1048 ms. Trace:
I20260620 19:58:55.675773 8001 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.676178 8340 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.677079 8070 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:55.681129 8328 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:55.691859 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:55.718995 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 89476.6us
W20260620 19:58:55.703872 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.631031 8039 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
W20260620 19:58:55.695611 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 53766.9us
I20260620 19:58:55.632512 8356 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20260620 19:58:55.734169 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.737514 8335 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:55.747622 8342 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:55.749487 8359 raft_consensus.cc:3055] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:55.767377 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:55.768308 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:55.772599 6345 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:55.732545 8215 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.732796 8146 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.753554 8144 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.754629 8251 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:58:55.801378 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:55.807762 6375 raft_consensus.cc:2393] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:55.750912 8337 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.814467 6232 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:55.816661 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:55.785799 8352 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.840570 6875 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:58:55.841212 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 75968.5us
W20260620 19:58:55.847669 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 91649.6us
I20260620 19:58:55.807512 8359 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:55.858920 6299 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:55.849324 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:55.864070 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:55.809118 8001 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.867308 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:55.867187 8146 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.867874 6195 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:55.868441 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:55.863860 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:55.869248 8146 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:55.818045 8144 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.869853 8338 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:55.869954 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:55.870812 8371 consensus_queue.cc:1059] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:55.871206 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:55.849715 8251 raft_consensus.cc:697] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:55.871747 6300 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:55.863888 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:55.873318 6231 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260620 19:58:55.850944 8330 log.cc:927] Time spent T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260620 19:58:55.879300 6381 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
W20260620 19:58:55.876308 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 85766.4us
I20260620 19:58:55.878743 6226 raft_consensus.cc:1275] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260620 19:58:55.882436 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:55.842471 7925 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:55.772050 6902 rpcz_store.cc:276] 0620 19:58:54.626186 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:54.645026 (+ 18840us) service_pool.cc:224] Handling call
0620 19:58:55.598877 (+953851us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:58:55.674738 (+ 75861us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19440896}
W20260620 19:58:55.879674 8331 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260620 19:58:55.873111 8224 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:55.871754 8335 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:55.878654 8058 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.849982 7933 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
W20260620 19:58:55.923266 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 123205us
W20260620 19:58:55.930315 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.910121 8052 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260620 19:58:55.935586 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 60504.9us
I20260620 19:58:55.864250 8215 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.912928 8367 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:55.875191 8352 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:55.944509 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 62141.5us
I20260620 19:58:55.905427 8001 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.946713 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1683) took 1200 ms. Trace:
I20260620 19:58:55.968449 8039 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:55.968680 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:55.968900 8039 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:55.969321 8039 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:55.970445 8144 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:55.941413 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 82270.6us
I20260620 19:58:55.975107 6375 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:58:55.976298 6345 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:55.978148 6875 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:55.939988 8052 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:58:55.975831 8273 consensus_meta.cc:287] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.871s user 0.005s sys 0.009s
I20260620 19:58:55.929666 8058 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.000061 8224 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:55.959887 6303 rpcz_store.cc:276] 0620 19:58:54.745988 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:54.799535 (+ 53547us) service_pool.cc:224] Handling call
0620 19:58:54.800888 (+ 1353us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:54.893040 (+ 92152us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:54.893089 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:54.910878 (+ 17789us) log.cc:844] Serialized 26 byte log entry
0620 19:58:55.153846 (+242968us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:58:55.170544 (+ 16698us) log.cc:844] Serialized 14 byte log entry
0620 19:58:55.173764 (+ 3220us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:55.175348 (+ 1584us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:55.667135 (+491787us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:58:55.938877 (+271742us) raft_consensus.cc:1685] finished
0620 19:58:55.946427 (+ 7550us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:55.946634 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91661,"prepare.queue_time_us":23293,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":92304,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":8453,"spinlock_wait_cycles":5321472,"thread_start_us":224119,"threads_started":1,"wal-append.queue_time_us":393887}
I20260620 19:58:55.908893 8251 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:56.002947 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:55.953624 8334 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:56.083473 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 105353us
I20260620 19:58:56.182657 6271 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:56.192144 6875 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:55.990541 8247 consensus_meta.cc:287] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.600s user 0.010s sys 0.000s
I20260620 19:58:56.001098 8176 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 19:58:56.087827 8369 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.211s user 0.000s sys 0.001s
W20260620 19:58:56.160411 8327 consensus_meta.cc:287] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.818s user 0.006s sys 0.006s
I20260620 19:58:56.193423 8176 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.193835 8327 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.194043 8176 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:55.974330 8366 raft_consensus.cc:697] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:58:56.189991 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 224237us
W20260620 19:58:56.102542 8345 consensus_meta.cc:287] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.524s user 0.010s sys 0.000s
W20260620 19:58:56.156136 8373 log.cc:927] Time spent T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260620 19:58:55.975242 8359 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.209069 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:56.159457 8328 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.165043 8267 consensus_queue.cc:1059] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260620 19:58:56.219465 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:56.156716 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:56.220849 8125 raft_consensus.cc:3055] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:58:56.222412 8125 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:56.224607 8267 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:55.993305 8052 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:55.994379 8215 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.211599 6303 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:56.155282 8270 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.156330 8390 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:55.975817 8367 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:56.002213 8346 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:58:56.195609 8290 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:56.225811 8058 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:56.167570 7933 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260620 19:58:56.238114 6345 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:56.051275 8273 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.219513 8387 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:56.238205 6223 raft_consensus.cc:2393] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:56.230046 8337 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.232221 8267 raft_consensus.cc:3055] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:56.244283 6302 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:58:56.234148 8390 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:56.240026 8383 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:58:56.243176 8359 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:56.261746 8352 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:56.270606 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 78529.9us
I20260620 19:58:56.244452 8355 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.252717 8267 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:56.256301 8390 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:56.238653 8270 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.274179 6344 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:56.240700 8290 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.258805 8365 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
W20260620 19:58:56.288424 6877 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.570s (SENT) [suppressed 26 similar messages]
I20260620 19:58:56.296931 8345 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:56.303332 8176 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:56.313088 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:56.317814 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:56.318333 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 80268us
I20260620 19:58:56.329890 8334 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.331099 8397 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:56.331256 7933 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.331679 8397 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:56.331935 7933 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.333642 8242 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.333789 8058 consensus_queue.cc:1059] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:56.334136 8242 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.336107 8242 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.337198 8039 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.339036 8327 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.339146 8150 raft_consensus.cc:3055] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:56.339695 8150 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:56.289354 8366 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.294446 8001 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.316447 8345 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:58:56.395346 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 55590.2us
I20260620 19:58:56.341150 8365 raft_consensus.cc:697] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:56.256567 8387 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:56.415221 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 140388us
I20260620 19:58:56.409237 8290 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.409255 8273 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:56.416553 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:56.410969 8112 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:56.411725 8378 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:56.395896 8383 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:58:56.413995 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 91002.6us
I20260620 19:58:56.421094 8378 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.421187 7933 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.422143 6345 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:58:56.422705 8001 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:58:56.423146 8215 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:56.423312 8378 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.423631 8215 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.424068 8215 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:56.405395 8176 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.424882 8176 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.426877 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:56.340632 8150 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:56.429086 8367 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:56.429095 8176 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:56.421236 8383 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.433178 8383 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.432623 8390 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.429136 6345 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:56.434350 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:56.411991 8242 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:56.406224 8345 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:56.438700 6306 raft_consensus.cc:2410] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:58:56.416977 8058 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.440768 6307 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260620 19:58:56.440644 8058 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.442850 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:56.443790 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:56.440023 8112 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.444909 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:56.443821 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:56.446826 6300 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:56.447454 8390 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.447716 8270 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.412145 8382 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.460608 6378 raft_consensus.cc:1240] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:58:56.448205 8112 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.463215 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.465058 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.465775 6378 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:56.466367 6372 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:56.466275 8365 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.465776 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:56.468444 8361 log.cc:927] Time spent T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.737s user 0.000s sys 0.004s
I20260620 19:58:56.442843 8058 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:56.452597 8001 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:56.506345 8399 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.507215 8399 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.507993 6228 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:58:56.510275 8058 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:56.504562 6376 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:56.463016 8039 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.512240 6273 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:56.448709 8242 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:56.515144 6875 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:56.414896 8267 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.519871 6273 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:56.521109 8399 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.523046 8112 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:56.525903 8112 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:56.526242 8112 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:56.526312 6376 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260620 19:58:56.528209 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.530952 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.531075 8058 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:56.531529 8387 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.531710 6372 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:56.531867 8058 raft_consensus.cc:697] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:56.513556 8270 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:58:56.534940 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 64964.8us
I20260620 19:58:56.520377 8385 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:56.460644 8382 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.545861 6875 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:56.534929 6380 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260620 19:58:56.536263 8406 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.542145 8385 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:56.561530 6381 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:56.563055 8058 consensus_queue.cc:237] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.538198 8270 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.566044 8270 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.568344 6376 raft_consensus.cc:1240] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:56.534726 8215 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.536496 8052 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.547989 7933 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260620 19:58:56.570792 8385 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:56.574265 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.596441 6227 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:56.576542 8273 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.591228 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:56.600601 6232 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:58:56.585739 8366 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260620 19:58:56.602217 6230 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:56.603461 8125 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.604633 8125 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:56.605868 8112 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.607537 8407 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.608090 8407 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.589999 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:56.590250 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:56.583524 6298 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:56.577364 8410 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:58:56.616381 6373 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:56.616882 8145 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:56.620683 8215 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.623520 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1688) took 1760 ms. Trace:
I20260620 19:58:56.650589 6304 rpcz_store.cc:276] 0620 19:58:54.862831 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:54.867825 (+ 4994us) service_pool.cc:224] Handling call
0620 19:58:54.887751 (+ 19926us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:58:54.935693 (+ 47942us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:54.935723 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:58:54.969221 (+ 33498us) log.cc:844] Serialized 532 byte log entry
0620 19:58:55.187069 (+217848us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:58:55.521908 (+334839us) log.cc:844] Serialized 14 byte log entry
0620 19:58:55.799684 (+277776us) log.cc:844] Serialized 14 byte log entry
0620 19:58:56.608758 (+809074us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b18003f66a0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b846185c 00007fa6b57abc8b 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c5c55 00007fa6b88b2440 00007fa6b88c4af6 00007fa6b88d5667
0620 19:58:56.608863 (+ 105us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:56.608935 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:56.623237 (+ 14302us) raft_consensus.cc:1685] finished
0620 19:58:56.623268 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:56.623449 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24481,"prepare.queue_time_us":34444,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":174632,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":7543,"spinlock_wait_cycles":20498688,"thread_start_us":522164,"threads_started":3,"wal-append.queue_time_us":348854}
I20260620 19:58:56.586138 8290 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:56.585839 8367 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.643317 6300 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:56.568351 8406 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:56.689728 8386 log.cc:927] Time spent T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.600s user 0.003s sys 0.000s
I20260620 19:58:56.645726 8052 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:56.596791 8327 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260620 19:58:56.611203 8273 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:56.755519 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:56.763439 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:56.890321 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:56.890797 6909 raft_consensus.cc:2410] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:58:56.886654 8363 raft_consensus.cc:3055] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:56.893086 8363 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:58:56.893208 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:56.890296 6908 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:56.894200 8363 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.902011 8363 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:56.824930 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.914311 6381 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:56.915714 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.760378 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.798166 8401 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.762885 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:56.925046 6874 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:56.926009 8058 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.762413 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.097617 6907 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 8.
I20260620 19:58:56.928593 8270 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.760267 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:56.691010 8145 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:58:56.758524 8382 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:56.940740 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 48429.3us
I20260620 19:58:56.690706 8052 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.757248 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:56.919446 7933 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.944737 8273 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:56.944847 6381 raft_consensus.cc:2393] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:56.750651 8367 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:56.945169 8398 raft_consensus.cc:3055] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:56.756417 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.927908 8406 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:56.758592 8290 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:56.761847 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:56.928855 8247 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:56.948606 8176 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:56.950807 8328 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:56.951536 6302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1696) took 1330 ms. Trace:
W20260620 19:58:56.958200 8403 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.487s user 0.000s sys 0.001s
I20260620 19:58:56.960323 6378 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:56.961393 6379 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:56.942575 6196 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:56.973522 8416 consensus_queue.cc:1059] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:56.977020 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:56.991493 8327 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.995806 8340 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:56.996850 8407 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:56.997757 8112 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:56.926364 8410 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260620 19:58:56.928184 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.109862 6906 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
W20260620 19:58:57.041556 8359 consensus_meta.cc:287] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.774s user 0.008s sys 0.003s
I20260620 19:58:57.061003 8397 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.064949 6126 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47092 (request call id 377) took 8661 ms. Trace:
I20260620 19:58:57.069626 6129 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47072 (request call id 394) took 8824 ms. Trace:
I20260620 19:58:57.105667 8058 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.076707 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.079896 8388 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.090787 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1820) took 1241 ms. Trace:
I20260620 19:58:57.093355 6380 raft_consensus.cc:1240] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:58:57.093633 6223 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1939) took 1304 ms. Trace:
I20260620 19:58:57.095688 6225 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260620 19:58:57.104064 8385 consensus_meta.cc:287] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.000s
I20260620 19:58:57.146098 6302 rpcz_store.cc:276] 0620 19:58:55.621863 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:55.623922 (+ 2059us) service_pool.cc:224] Handling call
0620 19:58:55.637711 (+ 13789us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:56.252835 (+615124us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:56.951261 (+698426us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b54001167d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:56.951466 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393540992}
I20260620 19:58:57.155326 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1821) took 1305 ms. Trace:
I20260620 19:58:57.105893 6903 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:57.148902 7933 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.147975 8052 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.144764 8398 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:57.142310 8335 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.014s
I20260620 19:58:57.156180 8290 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.155982 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.157313 6381 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:58:57.103758 8401 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.157692 8406 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.158681 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1941) took 1362 ms. Trace:
I20260620 19:58:57.143306 8365 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:57.029119 8345 consensus_meta.cc:287] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.575s user 0.007s sys 0.002s
W20260620 19:58:57.177484 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:57.221347 6224 rpcz_store.cc:276] 0620 19:58:55.796015 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:55.796141 (+ 126us) service_pool.cc:224] Handling call
0620 19:58:55.818387 (+ 22246us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:55.818591 (+ 204us) raft_consensus.cc:1534] Early marking committed up to index 2
0620 19:58:56.812909 (+994318us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b1800449aa0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b846185c 00007fa6b57abc8b 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c5c55 00007fa6bc2b4247 00007fa6b88c4d87 00007fa6b88d449f
0620 19:58:56.813107 (+ 198us) log.cc:844] Serialized 14 byte log entry
0620 19:58:57.158352 (+345245us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:57.158397 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:58:57.158436 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:57.158509 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:57.158625 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14408,"prepare.queue_time_us":128598,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"spinlock_wait_cycles":28087680,"thread_start_us":429092,"threads_started":3}
W20260620 19:58:57.074815 8391 log.cc:927] Time spent T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.657s user 0.002s sys 0.000s
I20260620 19:58:57.194118 6374 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:57.190778 6129 rpcz_store.cc:276] 0620 19:58:48.244786 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:48.327310 (+ 82524us) service_pool.cc:224] Handling call
0620 19:58:57.069393 (+8742083us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":295013,"reported_tablets":7}
I20260620 19:58:57.158764 8247 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.141789 8399 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260620 19:58:57.139420 8270 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.195942 6375 rpcz_store.cc:276] 0620 19:58:55.849400 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:55.849620 (+ 220us) service_pool.cc:224] Handling call
0620 19:58:55.872207 (+ 22587us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:55.975478 (+103271us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:57.090467 (+1114989us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400065254. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:57.090718 (+ 251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230607104}
I20260620 19:58:57.177841 6372 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:57.165707 8176 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.179967 8340 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.165024 8327 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.188616 8328 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.189320 6126 rpcz_store.cc:276] 0620 19:58:48.403795 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:48.435070 (+ 31275us) service_pool.cc:224] Handling call
0620 19:58:57.064835 (+8629765us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:58:54.759514 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:58:54.848663 (+ 89149us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:58:54.848732 (+ 69us) write_op.cc:435] Acquired schema lock
0620 19:58:54.848753 (+ 21us) tablet.cc:662] Decoding operations
0620 19:58:54.849607 (+ 854us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:58:54.849733 (+ 126us) write_op.cc:641] Partition lock acquired for write op
0620 19:58:54.849772 (+ 39us) tablet.cc:685] Acquiring locks for 4 operations
0620 19:58:54.850384 (+ 612us) tablet.cc:701] Row locks acquired
0620 19:58:54.850410 (+ 26us) write_op.cc:260] PREPARE: finished
0620 19:58:54.874342 (+ 23932us) write_op.cc:270] Start()
0620 19:58:54.893386 (+ 19044us) write_op.cc:276] Timestamp: P: 1781985534850541 usec, L: 0
0620 19:58:54.893413 (+ 27us) op_driver.cc:348] REPLICATION: starting
0620 19:58:54.940489 (+ 47076us) log.cc:844] Serialized 1686 byte log entry
0620 19:58:55.538972 (+598483us) op_driver.cc:464] REPLICATION: finished
0620 19:58:55.872050 (+333078us) write_op.cc:301] APPLY: starting
0620 19:58:55.872145 (+ 95us) tablet.cc:1367] starting BulkCheckPresence
0620 19:58:55.872295 (+ 150us) tablet.cc:1370] finished BulkCheckPresence
0620 19:58:55.916727 (+ 44432us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:58:55.957400 (+ 40673us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:58:55.957787 (+ 387us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0620 19:58:55.957853 (+ 66us) write_op.cc:312] APPLY: finished
0620 19:58:56.088115 (+130262us) log.cc:844] Serialized 41 byte log entry
0620 19:58:56.088307 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:58:56.088409 (+ 102us) write_op.cc:454] Released schema lock
0620 19:58:56.088462 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0620 19:58:56.088859 (+ 397us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:58:56.088894 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1564831,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":319880,"apply.run_cpu_time_us":7333,"apply.run_wall_time_us":216789,"num_ops":4,"prepare.queue_time_us":215331,"prepare.run_cpu_time_us":40480,"prepare.run_wall_time_us":424171,"raft.queue_time_us":63775,"raft.run_cpu_time_us":45397,"raft.run_wall_time_us":226982,"replication_time_us":643527,"thread_start_us":514205,"threads_started":4,"wal-append.queue_time_us":370891,"wal-append.run_cpu_time_us":9597,"wal-append.run_wall_time_us":1777060}]]}
I20260620 19:58:57.236610 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.211822 8416 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:58:57.203745 8388 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.258849 6381 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:57.205778 8415 consensus_queue.cc:1059] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260620 19:58:57.209221 8385 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.201464 6223 rpcz_store.cc:276] 0620 19:58:55.788929 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:55.794443 (+ 5514us) service_pool.cc:224] Handling call
0620 19:58:57.093366 (+1298923us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400077454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:57.093556 (+ 190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":773132672}
I20260620 19:58:57.307472 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.298791 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:57.309100 6374 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:57.309053 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.303828 8410 raft_consensus.cc:3055] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:58:57.311830 8410 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:57.312083 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.313643 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.319052 8052 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.319706 8052 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:58:57.320667 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:57.320267 8052 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:57.321154 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:57.322119 8411 log.cc:927] Time spent T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260620 19:58:57.322388 8416 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.247903 8290 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:57.238942 8345 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.205667 6379 rpcz_store.cc:276] 0620 19:58:55.850139 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:55.859536 (+ 9397us) service_pool.cc:224] Handling call
0620 19:58:56.960935 (+1101399us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b5800142e98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:58:56.961167 (+ 232us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:57.155065 (+193898us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:57.155271 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5915520}
I20260620 19:58:57.331326 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.276093 8359 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:57.235708 8406 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.239127 8365 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.334244 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.335853 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:57.339356 8417 log.cc:927] Time spent T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.107s user 0.002s sys 0.000s
I20260620 19:58:57.342351 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:57.314386 6373 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:57.348852 8242 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.349783 8340 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:57.351455 6229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1940) took 1562 ms. Trace:
I20260620 19:58:57.347313 6381 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 1.
I20260620 19:58:57.307761 6905 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:58:57.353168 6376 raft_consensus.cc:2393] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:57.360545 6229 rpcz_store.cc:276] 0620 19:58:55.789267 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:55.789388 (+ 121us) service_pool.cc:224] Handling call
0620 19:58:55.789667 (+ 279us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:55.812119 (+ 22452us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:55.812165 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:55.842348 (+ 30183us) log.cc:844] Serialized 26 byte log entry
0620 19:58:55.980572 (+138224us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:58:55.980948 (+ 376us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:55.981499 (+ 551us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:57.074047 (+1092548us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:58:57.344638 (+270591us) raft_consensus.cc:1685] finished
0620 19:58:57.344667 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:57.351373 (+ 6706us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1338,"raft.queue_time_us":1521,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":12339,"spinlock_wait_cycles":6148480,"thread_start_us":55482,"threads_started":1,"wal-append.queue_time_us":409107}
I20260620 19:58:57.361696 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:57.354027 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 383877us
I20260620 19:58:57.365465 6375 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:58:57.367564 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:57.368197 6375 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:57.369447 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.369488 6273 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:57.371052 6273 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:57.323140 8416 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.351822 8415 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.351926 8428 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:58:57.358618 8112 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:57.360517 8079 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:57.303175 8398 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.334457 8290 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.346081 8359 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.383745 6198 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:57.374485 8340 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:57.375336 8346 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260620 19:58:57.384887 8419 log.cc:927] Time spent T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.190s user 0.001s sys 0.000s
I20260620 19:58:57.367894 8242 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.021s
I20260620 19:58:57.386229 8382 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:57.418272 8382 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:57.418809 8382 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:57.385623 8267 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:57.396333 8079 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.427662 8406 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.428622 8416 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:57.430017 8345 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:57.431654 8365 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.434916 8052 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:57.397878 8367 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.436000 8327 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.393522 8125 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.437100 8125 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.437628 8327 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:57.438135 8327 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:57.387939 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.387213 8058 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.405658 7933 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.385421 8401 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.406353 6377 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:57.408687 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:57.382653 8415 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.414764 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.403419 8346 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:57.417687 8176 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.375192 8402 consensus_queue.cc:1059] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
W20260620 19:58:57.448861 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 523688us
W20260620 19:58:57.444197 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 501517us
I20260620 19:58:57.450198 6907 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:57.460585 6377 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:57.416843 8359 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.393182 8112 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.405279 8340 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.440367 6379 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:58:57.520798 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.620369 6908 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:57.618635 6877 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:57.621737 6877 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:57.440613 8422 raft_consensus.cc:3055] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
W20260620 19:58:57.526922 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:57.443759 8273 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.440680 8388 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.443843 8398 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:57.439725 8365 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:57.436012 8367 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.597515 8427 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.439575 8125 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.439355 8328 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.438575 8327 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:57.626202 8422 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:58:57.587572 8363 consensus_meta.cc:287] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.685s user 0.012s sys 0.000s
I20260620 19:58:57.443637 8247 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:57.442919 8290 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.639320 6877 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:57.633476 8429 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.440057 8385 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.519965 8052 consensus_queue.cc:1059] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.152s
I20260620 19:58:57.681232 8052 consensus_queue.cc:1243] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260620 19:58:57.635403 8426 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.458334 6374 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:57.600086 8079 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.600478 8425 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.603976 8415 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.535377 8406 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:58:57.692474 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:57.659540 6377 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:58:57.602944 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:57.484692 8346 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:57.703573 6377 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260620 19:58:57.673823 6908 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:58:57.640295 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 309) took 1059 ms. Trace:
I20260620 19:58:57.678496 8112 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:57.634126 8401 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:57.638832 8058 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 19:58:57.683645 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 297097us
I20260620 19:58:57.660550 6128 catalog_manager.cc:5696] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:58:57.687332 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.670828 8367 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.707625 6225 rpcz_store.cc:276] 0620 19:58:56.581325 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.581482 (+ 157us) service_pool.cc:224] Handling call
0620 19:58:57.095444 (+513962us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:57.201587 (+106143us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:57.640063 (+438476us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b5400077450. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:57.640225 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1149645568}
I20260620 19:58:57.640398 8273 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.675189 8416 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260620 19:58:57.712515 6910 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:57.731420 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:57.734161 6345 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:58:57.737267 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 364042us
I20260620 19:58:57.659886 8427 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.742558 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.743243 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.743414 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:57.743703 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.680431 8365 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:57.741853 6345 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:57.712836 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.755934 6377 raft_consensus.cc:2393] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:57.705900 8052 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:57.709270 6904 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:57.754417 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:57.744002 6302 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:57.757752 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.758117 6307 raft_consensus.cc:1240] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:57.761065 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.764050 6375 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:57.711381 8401 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:57.765832 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:57.744016 6379 raft_consensus.cc:2393] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:58:57.744954 6128 catalog_manager.cc:5696] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:58:57.767037 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 101598us
I20260620 19:58:57.767382 6303 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260620 19:58:57.767727 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:57.768057 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.769170 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:57.663398 8422 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.775007 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:57.769706 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:57.772886 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 133615us
I20260620 19:58:57.773113 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.777045 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.778256 6874 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:57.779696 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:57.779990 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:57.780608 6908 raft_consensus.cc:2393] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:57.750238 8247 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:57.782987 6305 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:57.746711 8428 raft_consensus.cc:697] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:57.692579 8426 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.785072 6128 catalog_manager.cc:5696] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:57.795468 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.796211 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.798947 6300 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
W20260620 19:58:57.799481 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 415673us
I20260620 19:58:57.799902 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:57.800894 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:57.706547 8406 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.684731 8429 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.702965 8425 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:58.217883 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:58.061168 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 277848us
I20260620 19:58:58.244415 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:57.831983 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:58.093086 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:58.245703 6304 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:58:57.949831 6128 catalog_manager.cc:5696] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:58:57.968609 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:58.247830 6374 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 1.
I20260620 19:58:57.832481 6911 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260620 19:58:58.247053 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:58.100154 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 389) took 1532 ms. Trace:
I20260620 19:58:58.249034 6303 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:57.887518 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:58.249293 6378 rpcz_store.cc:276] 0620 19:58:56.567255 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.567375 (+ 120us) service_pool.cc:224] Handling call
0620 19:58:56.959843 (+392468us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b5800142e98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:58:56.960104 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:57.147132 (+187028us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:58.099846 (+952714us) spinlock_profiling.cc:243] Waited 96.3 ms on lock 0x7b54002f4ca0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88321a0 00007fa6b88bdde2 00007fa6b88ba1b6 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22
0620 19:58:58.100083 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223969408}
I20260620 19:58:58.250439 6305 raft_consensus.cc:2410] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260620 19:58:57.803333 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:58.088039 7933 raft_consensus.cc:3055] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
W20260620 19:58:58.066838 6907 consensus_meta.cc:287] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.570s user 0.008s sys 0.002s
I20260620 19:58:58.251245 6375 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
W20260620 19:58:58.012789 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 245062us
I20260620 19:58:57.960783 6196 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:58.251520 6907 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
W20260620 19:58:58.153388 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:58.097132 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1828) took 1503 ms. Trace:
I20260620 19:58:58.254803 6227 rpcz_store.cc:276] 0620 19:58:56.593855 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.685283 (+ 91428us) service_pool.cc:224] Handling call
0620 19:58:56.685562 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:57.187841 (+502279us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:57.187892 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:58:57.187952 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:57.188006 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:58.096853 (+908847us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b54001d3dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:58.096912 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:58.097066 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1137065728}
I20260620 19:58:57.766773 8401 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:57.764567 8052 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:58.252033 6345 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:58.250938 7933 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:58.244602 6875 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:58.260675 6345 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:57.859618 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:58.275336 8443 ts_tablet_manager.cc:933] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:58.299654 8447 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:58.300679 8447 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:58.301162 8447 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:58.304414 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:58.305178 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 344357us
I20260620 19:58:57.788244 8247 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:58:58.304879 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 56425us
I20260620 19:58:58.307895 6196 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:58.313875 8247 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:58.083287 8001 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.232s
I20260620 19:58:58.314455 6128 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 3, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:58:58.196874 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 391) took 1598 ms. Trace:
I20260620 19:58:58.316064 8273 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:58.316416 6380 rpcz_store.cc:276] 0620 19:58:56.598022 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.601146 (+ 3124us) service_pool.cc:224] Handling call
0620 19:58:57.093025 (+491879us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:57.204551 (+111526us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:58.196655 (+992104us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400065250. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:58.196811 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1129916288}
I20260620 19:58:58.316072 8001 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:58:58.091094 6905 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 734) took 1330 ms (client timeout 1561 ms). Trace:
I20260620 19:58:58.317624 8001 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:58:58.317988 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 452699us
W20260620 19:58:58.318097 6905 rpcz_store.cc:269] 0620 19:58:56.760573 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.761595 (+ 1022us) service_pool.cc:224] Handling call
0620 19:58:58.090809 (+1329214us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54002a3354. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:58.090987 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":361396992}
I20260620 19:58:58.318833 8436 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:58.319235 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:58.319568 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:58.319871 6304 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:58:58.320770 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:58.321350 6905 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
W20260620 19:58:58.083258 8413 log.cc:927] Time spent T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.784s user 0.000s sys 0.003s
I20260620 19:58:57.951454 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:58.198195 8327 consensus_meta.cc:287] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.538s user 0.006s sys 0.000s
I20260620 19:58:57.858778 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:58.319344 8436 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:58.316233 6128 catalog_manager.cc:5696] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 2, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 2 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:58:58.321570 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:58.724129 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 742) took 1261 ms. Trace:
I20260620 19:58:58.724720 8458 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:58.445554 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1484) took 1117 ms. Trace:
I20260620 19:58:58.725539 8458 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.725878 6373 rpcz_store.cc:276] 0620 19:58:57.315090 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.412118 (+ 97028us) service_pool.cc:224] Handling call
0620 19:58:57.412410 (+ 292us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:57.412867 (+ 457us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:57.412895 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:57.413541 (+ 646us) log.cc:844] Serialized 26 byte log entry
0620 19:58:57.762998 (+349457us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:58:57.767872 (+ 4874us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:57.777853 (+ 9981us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:58.252853 (+475000us) raft_consensus.cc:1685] finished
0620 19:58:58.260405 (+ 7552us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:58.432115 (+171710us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"thread_start_us":322968,"threads_started":1,"wal-append.queue_time_us":374368}
I20260620 19:58:58.496538 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 738) took 1124 ms. Trace:
I20260620 19:58:58.329144 6198 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:58:58.248808 8422 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:58:58.729127 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 468322us
W20260620 19:58:58.359510 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 51678.7us
I20260620 19:58:58.322988 7933 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.424978 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:57.802947 8176 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:58.743454 8176 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:58.744025 8176 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:58.743858 8401 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.746059 8401 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:58.746255 8365 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:58:58.746551 6877 leader_election.cc:336] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.807s (SENT)
I20260620 19:58:58.733253 8340 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:58.747057 6877 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:57.969549 8428 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.749912 8327 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.706770 8425 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:58.751112 8425 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:57.969115 8367 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:57.781183 8058 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:57.786080 8215 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:58:58.434557 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 730) took 1647 ms. Trace:
W20260620 19:58:58.532859 8398 consensus_meta.cc:287] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.894s user 0.009s sys 0.001s
W20260620 19:58:58.729811 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 400593us
I20260620 19:58:58.779484 6911 rpcz_store.cc:276] 0620 19:58:56.700046 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.701113 (+ 1067us) service_pool.cc:224] Handling call
0620 19:58:56.750868 (+ 49755us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:58.249255 (+1498387us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:58.347314 (+ 98059us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540068e2d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:58:58.347557 (+ 243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2485686656}
I20260620 19:58:58.724709 6377 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:58:58.780419 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:58.724715 6909 rpcz_store.cc:276] 0620 19:58:57.463151 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.463356 (+ 205us) service_pool.cc:224] Handling call
0620 19:58:57.496235 (+ 32879us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:57.519460 (+ 23225us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:57.519503 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:57.566016 (+ 46513us) log.cc:844] Serialized 26 byte log entry
0620 19:58:57.778210 (+212194us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:58:57.778249 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:57.778313 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:58.279687 (+501374us) raft_consensus.cc:1685] finished
0620 19:58:58.279715 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:58.724063 (+444348us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10177,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":18047,"spinlock_wait_cycles":1796352,"thread_start_us":144159,"threads_started":1,"wal-append.queue_time_us":681368}
W20260620 19:58:58.564505 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 245888us
W20260620 19:58:58.730310 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 485578us
I20260620 19:58:58.732507 8451 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:58.739411 8328 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:58.739912 8427 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:58.679651 8146 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:58.789125 8176 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.789279 8146 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.327365 8426 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:58.194518 8145 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
W20260620 19:58:58.332504 6904 consensus_meta.cc:287] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.551s user 0.008s sys 0.000s
I20260620 19:58:58.227419 8345 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:58.259439 8407 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:58.783011 8058 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.791188 6904 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:58.792348 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 744) took 1271 ms. Trace:
I20260620 19:58:58.793315 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:58.240967 8410 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:57.802780 8415 consensus_queue.cc:1059] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.939s
I20260620 19:58:57.952466 8430 consensus_queue.cc:1059] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.803s
I20260620 19:58:58.317538 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:58.713614 8406 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:58.742650 8150 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:58.742746 6298 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:58:58.808893 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:58.731568 6375 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:58:58.809907 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1844) took 1032 ms. Trace:
I20260620 19:58:58.810171 6375 rpcz_store.cc:276] 0620 19:58:57.777263 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.320109 (+542846us) service_pool.cc:224] Handling call
0620 19:58:58.731050 (+410941us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:58.809642 (+ 78592us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:58.809853 (+ 211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:58.792685 6904 rpcz_store.cc:276] 0620 19:58:57.520869 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.591509 (+ 70640us) service_pool.cc:224] Handling call
0620 19:58:58.792290 (+1200781us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:58.793592 8435 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:58.303138 8052 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:58.328616 8429 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:58.754251 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:58.772605 7933 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:58.739446 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:58:58.334724 8382 consensus_meta.cc:287] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.915s user 0.011s sys 0.000s
W20260620 19:58:58.340703 8346 consensus_meta.cc:287] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.631s user 0.006s sys 0.002s
I20260620 19:58:57.966742 8267 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:58.853243 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:58.853739 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:58.854154 6381 raft_consensus.cc:2410] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260620 19:58:58.433768 8434 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:58:58.481210 8371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.031s sys 0.041s
I20260620 19:58:58.734655 6345 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:58.782315 6875 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:58.785789 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:58.772006 8327 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:58.786279 8436 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:58.856155 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:58.742384 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:58.786517 8451 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.496544 8381 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260620 19:58:58.795469 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 476582us
I20260620 19:58:58.752048 8443 tablet_copy_client.cc:323] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.244.131:41701
I20260620 19:58:58.901585 6273 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:58.819612 8410 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:58.322269 8270 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:58.329531 8273 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.464s
I20260620 19:58:58.323832 8290 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:58:58.779541 8367 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:58:58.808095 6344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.008s sys 0.042s
I20260620 19:58:57.954383 8335 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:58:58.576856 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:58.910746 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 126451us
I20260620 19:58:58.990229 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 1846) took 1212 ms. Trace:
I20260620 19:58:58.811443 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 560) took 1124 ms. Trace:
I20260620 19:58:58.776413 6906 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:58:58.728266 8447 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.728840 6907 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:58:58.816417 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:58.329386 8439 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.479s
I20260620 19:58:58.798717 8398 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:58:58.583806 8432 consensus_queue.cc:1059] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.231s
W20260620 19:58:58.856812 6877 leader_election.cc:336] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.981s (SENT)
W20260620 19:58:58.901593 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 167621us
I20260620 19:58:58.819000 6903 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260620 19:58:58.858665 8371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.102s user 0.038s sys 0.041s
I20260620 19:58:58.742298 8247 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.905670 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:58.905706 6273 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:58.907238 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:58.798462 8345 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.810725 6379 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:58.814805 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 540) took 1075 ms. Trace:
I20260620 19:58:58.900671 8267 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.848801 8416 consensus_queue.cc:1059] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.202s
W20260620 19:58:58.933537 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 151234us
I20260620 19:58:58.995925 8335 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:58.995687 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 553) took 1310 ms. Trace:
I20260620 19:58:59.049067 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:58.791806 8407 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.049515 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:58:58.857652 8346 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.049810 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:59.048352 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:59.088217 6345 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:59.048560 6877 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:59.032207 6196 leader_election.cc:336] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 2.211s (SENT)
W20260620 19:58:59.089214 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 57043.1us
I20260620 19:58:58.990589 6376 rpcz_store.cc:276] 0620 19:58:57.777768 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.311012 (+533244us) service_pool.cc:224] Handling call
0620 19:58:58.990172 (+679160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:59.059137 8427 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:59.091007 6378 raft_consensus.cc:2368] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:58.991101 6909 rpcz_store.cc:276] 0620 19:58:57.687426 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.795108 (+1107682us) service_pool.cc:224] Handling call
0620 19:58:58.811400 (+ 16292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:59.052670 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:59.055358 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:59.057797 6374 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:58:59.061740 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.080356 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:59.066011 6232 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:59.093223 6227 raft_consensus.cc:2393] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 2: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:58.910748 8461 raft_consensus.cc:3055] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:58:59.103492 8461 raft_consensus.cc:740] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:58:59.066574 6300 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260620 19:58:58.988842 6344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s user 0.012s sys 0.044s
I20260620 19:58:58.934135 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.106876 6908 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 1.
I20260620 19:58:59.107161 8461 consensus_queue.cc:260] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:58.970985 8145 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.172s
I20260620 19:58:59.058241 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:58.904820 8429 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:58.972014 8001 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:58.931463 8340 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:58.968760 8328 raft_consensus.cc:697] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:58.911774 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:59.146210 6381 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:59.075017 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:59.075495 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 169828us
I20260620 19:58:58.991884 8454 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:59.075703 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.075996 7933 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:59.148672 7933 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:58.975378 8426 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:58.970377 8146 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.080521 6226 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260620 19:58:59.155673 6226 pending_rounds.cc:85] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 2
I20260620 19:58:59.156049 6226 pending_rounds.cc:107] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260620 19:58:59.156337 6226 raft_consensus.cc:2889] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:58:58.995947 6911 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:59.040395 8425 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:59.039419 8176 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:59.076352 6907 rpcz_store.cc:276] 0620 19:58:57.684861 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.304072 (+619211us) service_pool.cc:224] Handling call
0620 19:58:58.995633 (+691561us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:59.083830 6225 raft_consensus.cc:1240] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:58:59.158087 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.054471 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 567) took 1338 ms. Trace:
I20260620 19:58:59.092598 6229 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:58:58.852860 8382 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:59.090376 6196 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:59.162292 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 71959.2us
I20260620 19:58:59.095099 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.090653 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.095692 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.167209 6909 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.096552 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:58:59.097501 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.097319 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:59.097631 8381 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260620 19:58:59.098098 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 241227us
I20260620 19:58:59.093605 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.098680 8452 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:58:59.098793 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.065675 6298 rpcz_store.cc:276] 0620 19:58:57.738422 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.744294 (+ 5872us) service_pool.cc:224] Handling call
0620 19:58:58.813643 (+1069349us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:59.099463 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.099895 8456 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:59.100109 6345 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:59.100649 8365 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:59.058871 8367 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:59.096506 8427 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.102437 8398 consensus_queue.cc:1059] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260620 19:58:59.111352 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:58.970019 8460 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:59.081480 8335 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:58:59.118412 6344 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:58:59.118827 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 49216.3us
I20260620 19:58:59.119175 6301 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:58:59.119349 6377 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
W20260620 19:58:59.120448 8448 log.cc:927] Time spent T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260620 19:58:59.121236 8345 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.123651 8052 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.127223 6302 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:58:59.127679 8447 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:59.075269 6379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 1848) took 1286 ms (client timeout 1681 ms). Trace:
I20260620 19:58:59.227514 6307 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:58:59.133169 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:58:59.135150 6306 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:58:58.910303 8435 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.237753 6375 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:58:59.149149 7933 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:59.153942 8446 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:58:59.158666 6907 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:59.142369 8145 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:59.172462 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.174250 6230 raft_consensus.cc:2393] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:58:59.174109 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 74035.5us
I20260620 19:58:59.148434 8454 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:58:59.227767 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.178614 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.172312 8273 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:59.229197 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:59.230796 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 112439us
I20260620 19:58:59.160324 8371 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:58:59.232028 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 1847) took 1454 ms. Trace:
I20260620 19:58:59.232699 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.233469 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:59.234225 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:59.235989 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 572) took 1465 ms. Trace:
I20260620 19:58:59.170518 8381 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:59.229243 8215 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:58:59.237999 6198 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:59.286123 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:59.339448 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 101514us
I20260620 19:58:59.340260 6198 leader_election.cc:400] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:58:59.340497 6198 leader_election.cc:403] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260620 19:58:59.147037 8340 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:59.327776 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.346695 6909 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:59.347777 6198 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:59.157338 8429 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:59.160071 8426 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.339470 6231 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:59.383574 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.382381 6300 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:59.340631 8447 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:59.233168 8001 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:59.340523 8381 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.386054 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.334326 6911 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:59.333387 6303 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:59.227130 8365 raft_consensus.cc:697] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:59.284432 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:58.728257 6902 rpcz_store.cc:276] 0620 19:58:57.375782 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.382643 (+ 6861us) service_pool.cc:224] Handling call
0620 19:58:57.403708 (+ 21065us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:58:57.516413 (+112705us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:57.518223 (+ 1810us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:58:57.597611 (+ 79388us) log.cc:844] Serialized 26 byte log entry
0620 19:58:57.949226 (+351615us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:58:57.951686 (+ 2460us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:58:57.965941 (+ 14255us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:58:58.496217 (+530276us) raft_consensus.cc:1685] finished
0620 19:58:58.496255 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:58:58.496460 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50103,"raft.queue_time_us":60434,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11750,"thread_start_us":277729,"threads_started":1,"wal-append.queue_time_us":895490}
I20260620 19:58:59.167538 8425 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.329082 8454 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.333850 6908 rpcz_store.cc:276] 0620 19:58:57.770720 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.308806 (+538086us) service_pool.cc:224] Handling call
0620 19:58:59.235778 (+926972us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54006e0354. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:59.235916 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":594543360}
I20260620 19:58:59.175014 8397 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.153s
I20260620 19:58:59.159489 6906 rpcz_store.cc:276] 0620 19:58:57.711061 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.937271 (+226210us) service_pool.cc:224] Handling call
0620 19:58:59.049874 (+1112603us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:59.178941 8457 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:58:59.337142 6223 raft_consensus.cc:2410] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:58:59.234884 8058 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:59.173650 8176 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:58:59.353437 6874 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:58:59.286161 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:59.388290 6907 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:58:59.332495 6377 rpcz_store.cc:276] 0620 19:58:57.778002 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.321336 (+543334us) service_pool.cc:224] Handling call
0620 19:58:59.231950 (+910614us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:59.333027 8273 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:59.389698 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:59.389374 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.416097 6911 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.393280 7933 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:58:59.418043 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 64659.7us
I20260620 19:58:59.399533 6232 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
W20260620 19:58:59.229691 6379 rpcz_store.cc:269] 0620 19:58:57.781728 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.259750 (+478022us) service_pool.cc:224] Handling call
0620 19:58:59.068259 (+808509us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:58:59.400367 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.340315 8371 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.398533 8467 consensus_queue.cc:1059] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260620 19:58:59.427894 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.428211 8467 raft_consensus.cc:3055] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:58:59.428732 8467 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:58:59.428974 8371 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.404798 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:59.429670 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.430320 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.430951 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.428978 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.339272 8328 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.432657 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.433279 6902 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:58:59.434131 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.386377 8327 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.397725 6905 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:58:59.402086 8474 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:59.408632 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.351101 8465 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:58:59.334260 8145 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:59.349596 8146 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260620 19:58:59.338196 8436 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260620 19:58:59.418129 7933 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.388350 6875 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:59.420532 6301 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1442) took 2971 ms (client timeout 1671 ms). Trace:
I20260620 19:58:59.385344 8447 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:59.402107 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:58:59.419068 8273 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:59.424213 6231 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:59.431478 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.386940 8270 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:59.438620 6911 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:58:59.499440 6228 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
W20260620 19:58:59.443559 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.406535 8457 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:59.440686 6195 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:58:59.500888 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 112568us
I20260620 19:58:59.411569 8267 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:59.476471 8052 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:58:59.489140 6301 rpcz_store.cc:269] 0620 19:58:56.442082 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.442631 (+ 549us) service_pool.cc:224] Handling call
0620 19:58:59.413530 (+2970899us) spinlock_profiling.cc:243] Waited 2.66 s on lock 0x7b54001a9354. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:58:59.413639 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6113192832}
I20260620 19:58:59.416224 8458 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.501705 6230 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:59.438158 8058 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260620 19:58:59.501566 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 60908us
I20260620 19:58:59.504469 6196 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:58:59.509305 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 178966us
I20260620 19:58:59.510007 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:58:59.510769 6196 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:59.437000 8346 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.423390 8460 raft_consensus.cc:697] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:58:59.435361 8365 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.503391 6305 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:58:59.483438 8465 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:59.506500 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:58:59.507544 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 179192us
I20260620 19:58:59.500856 8270 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:58:59.516306 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 297696us
I20260620 19:58:59.483283 6902 raft_consensus.cc:2376] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8. Re-sending same reply.
I20260620 19:58:59.419906 8176 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.519966 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.528198 6345 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:59.505200 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.532791 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.509553 8058 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:58:59.528671 6877 leader_election.cc:336] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.086s (SENT)
W20260620 19:58:59.533916 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.538347 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.516423 8428 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:58:59.503620 8457 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:58:59.533378 8052 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:59.534584 6344 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:59.556298 6344 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:59.550570 8058 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.529253 8270 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:59.559755 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.560935 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.561426 6911 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:59.536856 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.563023 6223 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:59.562954 8052 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.564054 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.547166 6877 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:58:59.528053 8422 consensus_meta.cc:287] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.756s user 0.006s sys 0.004s
I20260620 19:58:59.509248 8446 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:58:59.585088 8446 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:59.585469 8446 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:58:59.585876 8270 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.506393 6298 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:58:59.519061 8469 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:58:59.550969 8367 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:59.588245 8469 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:58:59.588702 8469 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:58:59.519965 8458 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:59.590866 8458 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.529290 8465 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:58:59.429674 8467 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.488570 8474 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.564815 6223 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:59.559759 8340 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.583973 8052 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:59.547345 6230 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:59.588485 8367 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:58:59.520316 8439 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:58:59.539407 8382 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:59.600317 6910 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:58:59.604213 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:58:59.662966 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 146388us
I20260620 19:58:59.534685 8452 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260620 19:58:59.592970 8458 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:59.604473 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 103675us
I20260620 19:58:59.567061 8479 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:58:59.672416 6907 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:58:59.665117 6195 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:59.670568 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:59.675597 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 139778us
I20260620 19:58:59.622975 8422 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:58:59.677292 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 121020us
W20260620 19:58:59.678207 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.664219 8415 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:59.662474 8439 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:59.668097 8340 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:58:59.675340 8467 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.672967 8345 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:58:59.695341 8458 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:58:59.675158 8429 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:59.736539 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:58:59.731400 6345 connection.cc:256] RPC call timeout handler was delayed by -1.5156556940000883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260620 19:58:59.745820 6345 connection.cc:256] RPC call timeout handler was delayed by -1.4598556940001117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1623s for a response
W20260620 19:58:59.746171 6345 connection.cc:256] RPC call timeout handler was delayed by -1.4400556940000797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
I20260620 19:58:59.737345 6230 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
W20260620 19:58:59.746508 6345 connection.cc:256] RPC call timeout handler was delayed by -1.4337556940000695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20260620 19:58:59.747854 6345 connection.cc:256] RPC call timeout handler was delayed by -1.2033556940000381s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
I20260620 19:58:59.711174 8479 raft_consensus.cc:697] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:58:59.772266 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.718150 8439 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:58:59.781363 6195 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:58:59.720305 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:58:59.786556 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:58:59.789624 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:58:59.796737 6877 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:59.735924 8458 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:58:59.808763 6877 leader_election.cc:304] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:59.809979 6911 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:58:59.796783 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.832820 8398 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:59.814471 8458 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.813100 8340 raft_consensus.cc:697] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:58:59.811201 8484 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:59.807328 6906 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
W20260620 19:58:59.838965 8497 log.cc:927] Time spent T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.061s user 0.000s sys 0.002s
I20260620 19:58:59.723999 8415 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.833230 8499 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:58:59.778578 8426 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:58:59.842803 6877 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:59.883201 6877 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:58:59.779948 8429 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:58:59.746373 8345 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.627388 8460 consensus_queue.cc:237] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.795338 8145 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:59.796562 8365 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:58:59.857412 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:58:59.859010 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:58:59.841862 8176 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:58:59.843487 8270 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:58:59.885262 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 136602us
I20260620 19:58:59.890333 6911 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:58:59.890295 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:58:59.887646 8052 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:58:59.892319 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 93733.7us
I20260620 19:58:59.924875 8340 consensus_queue.cc:237] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:58:59.934983 6195 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:58:59.930147 6377 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:58:59.938040 6380 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260620 19:58:59.897279 6230 raft_consensus.cc:2393] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:58:59.882514 8447 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:58:59.940912 8496 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:58:59.888023 8398 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:58:59.965483 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:59.966388 6377 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
W20260620 19:58:59.967664 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 77445.1us
I20260620 19:58:59.972381 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:58:59.984346 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:00.046057 8476 log.cc:927] Time spent T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
I20260620 19:59:00.050551 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:58:59.951683 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:58:59.898679 8365 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:58:59.981030 8461 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:00.062942 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:00.065857 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:00.072897 6390 tablet_copy_service.cc:140] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Received BeginTabletCopySession request for tablet 42a982b7017a41e1b0ca419a82cacf59 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 ({username='slave'} at 127.0.0.1:46186)
I20260620 19:59:00.078449 6390 tablet_copy_service.cc:161] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Beginning new tablet copy session on tablet 42a982b7017a41e1b0ca419a82cacf59 from peer 20bd146dc0cb4917a1a5ce05ec609ce1 at {username='slave'} at 127.0.0.1:46186: session id = 20bd146dc0cb4917a1a5ce05ec609ce1-42a982b7017a41e1b0ca419a82cacf59
I20260620 19:59:00.079296 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.080754 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.084501 8247 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.085525 8491 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:00.086119 8430 raft_consensus.cc:3055] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:00.086510 8430 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:00.047986 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.081422 6229 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 8.
W20260620 19:59:00.135061 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 200193us
I20260620 19:59:00.137112 6195 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.138417 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.138700 6195 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:00.139811 6229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1955) took 1018 ms. Trace:
I20260620 19:59:00.140362 6229 rpcz_store.cc:276] 0620 19:58:59.121681 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.138076 (+1016395us) service_pool.cc:224] Handling call
0620 19:59:00.139734 (+ 1658us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:00.088817 8328 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.052s sys 0.094s
I20260620 19:59:00.143347 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:00.144356 6229 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.148438 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.154564 6911 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:00.159376 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.145956 8426 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:00.156174 8483 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:00.101693 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.165100 8493 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 19:59:00.146853 8328 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.052s sys 0.094s
I20260620 19:59:00.169221 6877 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.169956 6195 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:00.171885 8465 consensus_meta.cc:287] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.545s user 0.009s sys 0.005s
I20260620 19:59:00.172962 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.174033 8052 consensus_queue.cc:1059] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:00.175071 8328 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:00.175943 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1946) took 1110 ms. Trace:
I20260620 19:59:00.173703 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 575) took 1095 ms. Trace:
I20260620 19:59:00.176307 6227 rpcz_store.cc:276] 0620 19:58:59.065591 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.175246 (+1109655us) service_pool.cc:224] Handling call
0620 19:59:00.175518 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.175680 (+ 162us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.175701 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.175736 (+ 35us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:00.175769 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.175815 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.175896 (+ 81us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.172292 8465 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.177949 6227 raft_consensus.cc:1240] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:58:59.895665 8479 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.179463 8465 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.179597 6875 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:00.176358 6195 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.181010 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1949) took 1105 ms. Trace:
I20260620 19:59:00.181143 6875 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:00.181372 6232 rpcz_store.cc:276] 0620 19:58:59.075620 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.093394 (+ 17774us) service_pool.cc:224] Handling call
0620 19:59:00.180869 (+1087475us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54001d56d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.180958 (+ 89us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":525407616}
I20260620 19:59:00.182181 8176 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:00.183459 8176 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:00.183980 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.185124 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.185810 6378 raft_consensus.cc:2393] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:00.188750 6910 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 566) took 2479 ms (client timeout 1956 ms). Trace:
W20260620 19:59:00.189131 6910 rpcz_store.cc:269] 0620 19:58:57.709725 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.774783 (+ 65058us) service_pool.cc:224] Handling call
0620 19:59:00.188586 (+2413803us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54004bdcd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.188682 (+ 96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4164469504}
I20260620 19:59:00.190107 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.190663 6910 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:00.190723 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.191495 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.191915 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.192495 6902 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:00.193404 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.194005 6902 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:00.196275 8435 leader_election.cc:290] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.197803 8435 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:00.198130 8435 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:00.198436 8435 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:00.199652 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 783) took 1146 ms. Trace:
I20260620 19:59:00.200002 6907 rpcz_store.cc:276] 0620 19:58:59.052906 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.428776 (+375870us) service_pool.cc:224] Handling call
0620 19:59:00.199493 (+770717us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b5400176bd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.199598 (+ 105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551670144}
I20260620 19:59:00.140902 8491 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:00.164067 8485 raft_consensus.cc:2764] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260620 19:59:00.174091 8398 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:00.176723 6911 rpcz_store.cc:276] 0620 19:58:59.078203 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.160277 (+1082074us) service_pool.cc:224] Handling call
0620 19:59:00.173648 (+ 13371us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.178033 6230 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.089038 6379 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:00.161999 6376 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260620 19:59:00.156734 8486 log.cc:927] Time spent T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260620 19:59:00.163480 8415 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:00.203329 6381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1851) took 1039 ms. Trace:
I20260620 19:59:00.186530 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.235149 6378 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:00.220023 8454 consensus_meta.cc:287] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.812s user 0.026s sys 0.026s
I20260620 19:59:00.182919 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1948) took 1112 ms. Trace:
I20260620 19:59:00.238881 6232 rpcz_store.cc:276] 0620 19:58:59.070071 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.182212 (+1112141us) service_pool.cc:224] Handling call
0620 19:59:00.182453 (+ 241us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.182642 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.182664 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.182702 (+ 38us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:00.182730 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.182785 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.182871 (+ 86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.240427 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.241191 6232 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:00.242297 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.205834 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:00.214697 6345 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.246650 6345 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.247434 6345 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:00.219540 8426 raft_consensus.cc:697] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:00.231213 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.157660 8474 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:00.203948 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1947) took 1138 ms. Trace:
I20260620 19:59:00.223824 8398 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:00.229278 6377 raft_consensus.cc:1240] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
W20260620 19:59:00.234699 8469 consensus_meta.cc:287] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.645s user 0.020s sys 0.012s
I20260620 19:59:00.273165 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.273573 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.261029 6227 rpcz_store.cc:276] 0620 19:58:59.065854 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.177566 (+1111712us) service_pool.cc:224] Handling call
0620 19:59:00.177808 (+ 242us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.203779 (+ 25971us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.203911 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:00.243975 8381 consensus_meta.cc:287] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.850s user 0.016s sys 0.021s
I20260620 19:59:00.274009 6911 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:00.233223 6381 rpcz_store.cc:276] 0620 19:58:59.164209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.164835 (+ 626us) service_pool.cc:224] Handling call
0620 19:58:59.165202 (+ 367us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:59.165449 (+ 247us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:58:59.528294 (+362845us) log.cc:844] Serialized 14 byte log entry
0620 19:59:00.174130 (+645836us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b1800340660. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b846185c 00007fa6b57abc8b 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b84595da
0620 19:59:00.174318 (+ 188us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.174366 (+ 48us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:00.174416 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.174477 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.203270 (+ 28793us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44,"prepare.queue_time_us":402248,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":90,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":58099584,"thread_start_us":476575,"threads_started":2,"wal-append.queue_time_us":210984}
I20260620 19:59:00.275744 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1952) took 1155 ms. Trace:
I20260620 19:59:00.276650 6877 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.276522 6227 rpcz_store.cc:276] 0620 19:58:59.120835 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.274999 (+1154164us) service_pool.cc:224] Handling call
0620 19:59:00.275252 (+ 253us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.275445 (+ 193us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.275473 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.275516 (+ 43us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:00.275553 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.275606 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.275696 (+ 90us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.278774 6227 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:59:00.279331 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1838) took 1129 ms. Trace:
I20260620 19:59:00.279683 6227 rpcz_store.cc:276] 0620 19:58:59.149802 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.278211 (+1128409us) service_pool.cc:224] Handling call
0620 19:59:00.278548 (+ 337us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.279143 (+ 595us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.279287 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.260828 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.264066 8247 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:00.285175 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.285297 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.287489 6911 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:00.287424 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1841) took 1136 ms. Trace:
I20260620 19:59:00.289189 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.287387 6381 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:00.248600 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.319706 6225 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:59:00.286738 8381 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.320896 6375 raft_consensus.cc:1240] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:59:00.321072 6225 raft_consensus.cc:1240] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:00.321692 8381 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.285856 8474 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:00.322841 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1500) took 1086 ms. Trace:
I20260620 19:59:00.323275 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.323889 6911 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:00.323395 6375 rpcz_store.cc:276] 0620 19:58:59.236830 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.331331 (+ 94501us) service_pool.cc:224] Handling call
0620 19:58:59.331708 (+ 377us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:00.321355 (+989647us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.322558 (+ 1203us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54000230d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:00.322782 (+ 224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2273468288}
I20260620 19:59:00.263286 8455 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:00.293376 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 578) took 1214 ms. Trace:
I20260620 19:59:00.325795 6877 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:59:00.294317 8439 consensus_meta.cc:287] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.503s user 0.011s sys 0.005s
I20260620 19:59:00.294687 8427 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:00.295502 8410 consensus_meta.cc:287] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 1.256s user 0.014s sys 0.006s
I20260620 19:59:00.296902 6196 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.299007 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1843) took 1147 ms. Trace:
W20260620 19:59:00.300078 8446 consensus_meta.cc:287] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.714s user 0.021s sys 0.030s
I20260620 19:59:00.263692 8146 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:00.302902 6307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1708) took 1253 ms. Trace:
I20260620 19:59:00.336468 6307 rpcz_store.cc:276] 0620 19:58:59.049148 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.049604 (+ 456us) service_pool.cc:224] Handling call
0620 19:59:00.302754 (+1253150us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54003ed5d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.302847 (+ 93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":322889088}
I20260620 19:59:00.337141 8407 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:00.303931 8456 raft_consensus.cc:3055] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:00.298511 6390 tablet_copy_source_session.cc:215] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:59:00.307065 6228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 322) took 1090 ms. Trace:
I20260620 19:59:00.307723 8365 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.266870 8398 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:00.309417 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.310583 6224 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 5. Ops: []
W20260620 19:59:00.311334 8001 consensus_meta.cc:287] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.926s user 0.011s sys 0.000s
I20260620 19:59:00.290963 6232 rpcz_store.cc:276] 0620 19:58:59.150579 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.286574 (+1135995us) service_pool.cc:224] Handling call
0620 19:59:00.286853 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.287058 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.287094 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.287155 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:00.287203 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.287249 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.287360 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.338024 8406 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:00.312395 8451 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:00.313045 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.355096 6376 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:59:00.355965 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 154737us
I20260620 19:59:00.313535 6378 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:00.314386 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1710) took 1253 ms. Trace:
I20260620 19:59:00.358011 6305 rpcz_store.cc:276] 0620 19:58:59.060550 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.097085 (+ 36535us) service_pool.cc:224] Handling call
0620 19:59:00.314233 (+1217148us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b54001cf054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.314330 (+ 97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":767496064}
I20260620 19:59:00.316311 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:00.317260 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:00.319275 8335 consensus_meta.cc:287] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 1.035s user 0.024s sys 0.035s
I20260620 19:59:00.292222 6345 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.273052 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 737) took 2906 ms. Trace:
W20260620 19:59:00.355394 8058 consensus_meta.cc:287] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.796s user 0.009s sys 0.010s
I20260620 19:59:00.325757 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.328540 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.223704 8430 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.334909 6225 raft_consensus.cc:1240] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260620 19:59:00.333603 6230 rpcz_store.cc:276] 0620 19:58:59.151079 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.298205 (+1147126us) service_pool.cc:224] Handling call
0620 19:59:00.298521 (+ 316us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.298705 (+ 184us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.298730 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.298769 (+ 39us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:00.298801 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.298854 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.298958 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.335302 6196 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.335757 8496 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:00.336572 8146 raft_consensus.cc:697] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:00.306106 6226 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:00.287205 8247 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:00.338290 6911 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:00.252686 8454 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:00.344063 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:00.340176 6228 rpcz_store.cc:276] 0620 19:58:59.217271 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.397898 (+180627us) service_pool.cc:224] Handling call
0620 19:59:00.306907 (+909009us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b54000b6654. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.307013 (+ 106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48861952}
I20260620 19:59:00.327997 8455 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:00.331056 8474 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:00.283581 8469 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:00.359542 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:00.339092 8456 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:00.357537 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.363236 8340 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:00.380388 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:00.360746 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 68577.5us
I20260620 19:59:00.326393 6907 rpcz_store.cc:276] 0620 19:58:59.078858 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.288898 (+1210040us) service_pool.cc:224] Handling call
0620 19:59:00.293317 (+ 4419us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.340044 6372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1853) took 1168 ms. Trace:
I20260620 19:59:00.352380 8345 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.332507 8504 consensus_queue.cc:1059] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
W20260620 19:59:00.357434 8273 consensus_meta.cc:287] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.848s user 0.000s sys 0.018s
I20260620 19:59:00.427944 6877 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.356103 6379 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:00.426441 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:00.430042 6304 raft_consensus.cc:2376] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2. Re-sending same reply.
I20260620 19:59:00.369285 6196 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:00.355579 8001 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:59:00.367488 6226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1844) took 1216 ms. Trace:
I20260620 19:59:00.332296 8439 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.439216 6378 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260620 19:59:00.417424 8430 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:00.337646 8446 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.382989 6301 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260620 19:59:00.460109 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.460821 6875 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:00.461241 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.462191 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.463094 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:00.402153 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:00.423619 6909 raft_consensus.cc:2368] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:59:00.371214 8270 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:00.468951 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.379315 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.416095 6307 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:59:00.422719 8340 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:00.372432 8247 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:00.375339 8474 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:00.327417 8458 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.358517 8406 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.428936 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.432390 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.435544 6231 raft_consensus.cc:2393] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 2: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.375559 8455 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:00.436270 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 67054.6us
I20260620 19:59:00.508368 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.458802 8362 raft_consensus.cc:3055] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:00.427671 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.510668 6230 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.438154 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.371016 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 776) took 1578 ms. Trace:
I20260620 19:59:00.435240 6226 rpcz_store.cc:276] 0620 19:58:59.151328 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.305497 (+1154169us) service_pool.cc:224] Handling call
0620 19:59:00.305803 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.367249 (+ 61446us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.367428 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.441730 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 554) took 2756 ms. Trace:
I20260620 19:59:00.511286 6911 rpcz_store.cc:276] 0620 19:58:58.792102 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.323063 (+1530961us) service_pool.cc:224] Handling call
0620 19:59:00.370957 (+ 47894us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.511525 6907 rpcz_store.cc:276] 0620 19:58:57.685494 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.434073 (+2748579us) service_pool.cc:224] Handling call
0620 19:59:00.434427 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.441296 (+ 6869us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.441338 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.441397 (+ 59us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:00.441442 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.441518 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.441660 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.369904 8058 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.444456 8434 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:00.444628 8484 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:00.444959 8410 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:00.422438 6372 rpcz_store.cc:276] 0620 19:58:59.169912 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.171164 (+ 1252us) service_pool.cc:224] Handling call
0620 19:58:59.282798 (+111634us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:59.338892 (+ 56094us) raft_consensus.cc:1534] Early marking committed up to index 3
0620 19:58:59.791293 (+452401us) log.cc:844] Serialized 14 byte log entry
0620 19:59:00.284264 (+492971us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.292982 (+ 8718us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:00.323671 (+ 30689us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.330362 (+ 6691us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.338718 (+ 8356us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4973,"prepare.queue_time_us":320329,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":22650,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":22270,"thread_start_us":709929,"threads_started":2,"wal-append.queue_time_us":544181}
I20260620 19:59:00.446916 6229 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.361267 6903 rpcz_store.cc:276] 0620 19:58:57.366884 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.373489 (+ 6605us) service_pool.cc:224] Handling call
0620 19:58:57.384584 (+ 11095us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:58:59.056085 (+1671501us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.262575 (+1206490us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400740254. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:00.272952 (+ 10377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3238779264}
I20260620 19:59:00.449388 8435 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:00.450156 8490 log.cc:927] Time spent T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.611s user 0.000s sys 0.007s
I20260620 19:59:00.540892 8454 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:00.543923 8458 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.451277 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.453123 8469 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:00.439410 8477 log.cc:927] Time spent T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260620 19:59:00.454310 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:00.372697 8496 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:00.547019 6304 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:00.547547 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1849) took 2765 ms. Trace:
I20260620 19:59:00.547384 8496 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.547894 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.548656 6304 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:00.549391 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.549818 8425 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:00.549903 6304 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:00.426906 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.551023 6300 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:00.365159 8426 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.473423 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.515082 8474 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:00.505882 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:00.468701 8467 consensus_meta.cc:287] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.737s user 0.008s sys 0.002s
I20260620 19:59:00.476649 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.509675 6196 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.363250 8335 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.564668 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.565267 8446 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:00.565774 8335 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:00.447530 8505 log.cc:927] Time spent T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260620 19:59:00.512327 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.566244 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.566704 6911 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:59:00.473824 8340 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:00.567474 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.518214 8410 raft_consensus.cc:697] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:00.505152 8483 consensus_queue.cc:1059] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260620 19:59:00.517179 8484 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.568544 8410 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.514473 8456 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.570087 8484 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:00.518214 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.436992 8273 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.523869 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.526134 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:00.527201 6306 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 546) took 2773 ms (client timeout 2886 ms). Trace:
I20260620 19:59:00.527300 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.531004 8435 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:00.476260 8466 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:00.510303 8362 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:00.533773 8457 consensus_meta.cc:287] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.967s user 0.006s sys 0.009s
W20260620 19:59:00.534164 8508 log.cc:927] Time spent T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20260620 19:59:00.534500 6902 raft_consensus.cc:2393] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:00.534821 6380 raft_consensus.cc:2368] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
I20260620 19:59:00.507123 8146 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:00.537839 8327 consensus_meta.cc:287] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.053s user 0.012s sys 0.004s
I20260620 19:59:00.546363 6305 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:00.474301 8427 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:00.554322 6302 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
W20260620 19:59:00.554836 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 45258.6us
I20260620 19:59:00.558252 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.565342 6225 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:00.567646 6223 raft_consensus.cc:2410] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:00.567898 8443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a982b7017a41e1b0ca419a82cacf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:59:00.571074 6911 raft_consensus.cc:2393] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:00.523636 6906 raft_consensus.cc:2410] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:00.518945 8058 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:00.515338 8434 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:00.519260 8381 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:00.573261 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.629834 6874 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:00.447522 8439 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.580662 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:00.596369 6306 rpcz_store.cc:269] 0620 19:58:57.753978 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.754280 (+ 302us) service_pool.cc:224] Handling call
0620 19:59:00.526959 (+2772679us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54001aa754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.527129 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3133006080}
I20260620 19:59:00.760159 6377 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:00.599602 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.600440 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.535168 8479 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:00.599602 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.725332 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.730408 6375 raft_consensus.cc:2393] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:00.576980 8407 consensus_queue.cc:1175] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" responder_term: 4 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4. Ops: [3.5-3.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260620 19:59:00.726687 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.729522 6196 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:00.638700 6298 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:00.597270 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:00.780797 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.646135 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 323) took 1262 ms. Trace:
I20260620 19:59:00.791131 6302 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:00.791399 6231 rpcz_store.cc:276] 0620 19:58:59.383932 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.505849 (+121917us) service_pool.cc:224] Handling call
0620 19:59:00.645937 (+1140088us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b5400372dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.646078 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2081863552}
I20260620 19:59:00.598863 6903 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 1 FOLLOWER]: Advancing to term 2
W20260620 19:59:00.790635 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 61131.2us
I20260620 19:59:00.761240 6378 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:00.761693 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 131925us
I20260620 19:59:00.742041 6307 raft_consensus.cc:2393] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 2: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:00.749666 6227 raft_consensus.cc:1240] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:59:00.802295 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:00.675900 6303 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:00.803754 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.804210 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:00.804745 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:00.763623 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:00.805444 6303 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:00.809620 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.814936 6304 raft_consensus.cc:2410] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260620 19:59:00.818639 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.819187 6911 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:00.818914 8362 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.820372 8362 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:00.547922 6373 rpcz_store.cc:276] 0620 19:58:57.782053 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.738826 (+956773us) service_pool.cc:224] Handling call
0620 19:58:58.739535 (+ 709us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:58:58.850298 (+110763us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:58:58.855467 (+ 5169us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:58:59.091329 (+235862us) log.cc:844] Serialized 52 byte log entry
0620 19:58:59.229607 (+138278us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:58:59.234759 (+ 5152us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:00.185202 (+950443us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b1800449f80. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:00.547197 (+361995us) spinlock_profiling.cc:243] Waited 54.5 ms on lock 0x7b5400065254. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:00.547254 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:00.547303 (+ 49us) raft_consensus.cc:1685] finished
0620 19:59:00.547318 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:00.547487 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7519,"raft.queue_time_us":12009,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":56127,"spinlock_wait_cycles":132707328,"thread_start_us":49546,"threads_started":1,"wal-append.queue_time_us":201887}
I20260620 19:59:00.790529 8479 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.779842 8425 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:59:00.825007 8425 raft_consensus.cc:697] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:00.825333 8479 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.825878 8425 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:00.640511 8514 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260620 19:59:00.746729 6198 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:00.827025 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:00.827564 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 80895.5us
I20260620 19:59:00.828230 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.828533 6198 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.704231 6379 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:00.792591 6229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1834) took 1728 ms. Trace:
I20260620 19:59:00.830322 6379 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:00.830127 6229 rpcz_store.cc:276] 0620 19:58:59.064669 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.205611 (+1140942us) service_pool.cc:224] Handling call
0620 19:59:00.792430 (+586819us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001d38d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.792541 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423603456}
I20260620 19:59:00.831367 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.832474 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.833248 6229 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.706552 8480 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:00.833606 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.834232 8435 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:00.834422 6372 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 9: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:00.834718 8435 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:00.834833 8465 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:59:00.835180 8435 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:00.773729 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.836603 6907 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:00.837456 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.837981 6907 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:00.800813 6225 raft_consensus.cc:2393] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:00.800866 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 44307us
I20260620 19:59:00.839753 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.840744 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.841706 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.842814 8001 consensus_queue.cc:1059] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:00.843178 8500 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:00.843637 8001 consensus_queue.cc:1059] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:00.833585 6911 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:00.833905 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.844921 6229 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:00.801259 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:00.844934 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.810092 6302 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:59:00.845947 6911 raft_consensus.cc:2410] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:00.846267 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:00.847023 6225 raft_consensus.cc:2393] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.644697 8328 raft_consensus.cc:3055] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:00.741725 8327 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:59:00.827630 6230 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:59:00.829726 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.833933 8480 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:00.794181 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.767506 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.571310 8469 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:00.761185 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:00.859639 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 50385.5us
I20260620 19:59:00.598402 6904 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:00.875100 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.817646 8496 raft_consensus.cc:2804] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260620 19:59:00.765668 6273 connection.cc:256] RPC call timeout handler was delayed by -1.289748851000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
I20260620 19:59:00.875648 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.850749 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.877821 6230 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:00.851553 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.878697 6302 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:00.643009 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.879935 6908 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:00.847981 6345 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.854106 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.806563 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:00.854854 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:00.731312 8247 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.852351 8328 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:00.622310 8454 raft_consensus.cc:3055] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:00.877350 6226 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:00.891170 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:00.852766 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.891976 6305 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:00.773965 8434 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:00.860849 6874 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:00.856277 6911 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1cb7a18e80e145f6913b635487750934"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:46646
W20260620 19:59:00.642028 6905 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 782) took 1737 ms (client timeout 1507 ms). Trace:
I20260620 19:59:00.622641 8461 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:00.848500 6196 leader_election.cc:304] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:00.893877 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 45440us
I20260620 19:59:00.897347 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.901424 6907 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 9.
I20260620 19:59:00.855947 8001 raft_consensus.cc:3055] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:00.884815 6377 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:00.629607 8466 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.891309 8436 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:00.881647 6229 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:00.889552 6273 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.853452 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.854710 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:00.893121 6911 raft_consensus.cc:606] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:00.923236 6911 raft_consensus.cc:614] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:00.892062 6231 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:00.891114 8454 raft_consensus.cc:740] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:00.766001 8456 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:00.907181 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 46396.9us
I20260620 19:59:00.915122 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:00.883805 8496 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:00.929214 6874 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:00.779242 8457 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.916383 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:00.908735 8469 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:00.905783 8273 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:00.914372 8427 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:00.966120 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:00.967096 6231 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:00.969930 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:00.917819 8001 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:00.973767 6374 raft_consensus.cc:1240] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:00.917106 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.916991 6271 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:00.856359 8381 raft_consensus.cc:3055] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:00.921154 6306 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:00.902617 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:00.924724 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:00.925431 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:00.960153 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:00.949524 8469 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:00.939865 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.916790 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:00.926385 6373 raft_consensus.cc:1240] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:00.889601 8500 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:00.977625 6907 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:00.979410 6911 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:00.949352 8410 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:00.913287 8461 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.941130 8496 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:00.926182 8516 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:59:00.980866 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:00.980858 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:01.005259 6225 raft_consensus.cc:2393] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:01.006093 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:00.965129 8365 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:00.981734 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:00.893586 6905 rpcz_store.cc:269] 0620 19:58:58.905012 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.063386 (+158374us) service_pool.cc:224] Handling call
0620 19:59:00.641815 (+1578429us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54005e2754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:00.641950 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":361000320}
W20260620 19:59:00.978662 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 148798us
I20260620 19:59:01.001722 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:01.009320 6907 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:01.010680 6196 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:00.976903 8381 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:00.978463 8469 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:01.012133 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:01.012761 6198 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:01.013123 6907 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:59:01.013846 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 557) took 3327 ms. Trace:
I20260620 19:59:01.011422 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:01.014106 6907 rpcz_store.cc:276] 0620 19:58:57.686452 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.012665 (+3326213us) service_pool.cc:224] Handling call
0620 19:59:01.012926 (+ 261us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:01.013639 (+ 713us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.013797 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:00.980943 8480 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:01.015470 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 100430us
I20260620 19:59:00.926527 8436 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:00.983141 8410 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:00.973802 8474 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.019469 6908 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:01.023447 8466 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.023224 8381 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.018007 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:01.029556 8435 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:01.006188 8500 raft_consensus.cc:697] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:01.014537 8367 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:59:01.014328 8516 raft_consensus.cc:697] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:01.031432 8430 consensus_meta.cc:287] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.572s user 0.007s sys 0.002s
W20260620 19:59:01.019124 8518 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260620 19:59:01.036429 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 417) took 1104 ms. Trace:
I20260620 19:59:01.063268 6380 rpcz_store.cc:276] 0620 19:58:59.932507 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.036086 (+103579us) service_pool.cc:224] Handling call
0620 19:59:01.017004 (+980918us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54000451d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:01.036353 (+ 19349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1028515072}
I20260620 19:59:01.038548 8410 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:01.032792 6381 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:01.017714 8273 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:01.015209 8365 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:01.034147 6305 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:00.949296 8328 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.039193 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 555) took 3353 ms. Trace:
I20260620 19:59:01.087355 6908 rpcz_store.cc:276] 0620 19:58:57.685859 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.009999 (+3324140us) service_pool.cc:224] Handling call
0620 19:59:01.019313 (+ 9314us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.038956 (+ 19643us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.039144 (+ 188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.070447 6877 leader_election.cc:400] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
I20260620 19:59:01.088536 6877 leader_election.cc:403] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260620 19:59:01.072171 8467 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:01.077663 8145 raft_consensus.cc:2764] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260620 19:59:01.017303 8454 consensus_queue.cc:260] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.079763 8436 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.079833 8365 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:01.083425 8521 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260620 19:59:01.084013 8500 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.012557 8340 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.097973 6232 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:01.063081 8381 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:01.103547 6231 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:01.071097 8466 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:01.112094 6908 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:46646
I20260620 19:59:01.070297 8367 raft_consensus.cc:697] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:01.112243 6232 raft_consensus.cc:1240] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260620 19:59:01.647272 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 335) took 1075 ms. Trace:
W20260620 19:59:01.106772 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 88836.2us
W20260620 19:59:01.241271 6344 leader_election.cc:336] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.730s (SENT)
W20260620 19:59:01.293408 6909 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 581) took 2214 ms (client timeout 1848 ms). Trace:
W20260620 19:59:01.648574 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 407224us
W20260620 19:59:01.425288 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:01.648036 6908 raft_consensus.cc:606] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:01.396260 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:01.648741 6231 rpcz_store.cc:276] 0620 19:59:00.571779 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.925246 (+353467us) service_pool.cc:224] Handling call
0620 19:59:01.647215 (+721969us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.099396 8430 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
W20260620 19:59:01.521049 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:01.654527 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:01.654892 6908 raft_consensus.cc:614] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:01.648278 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:01.647625 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1975) took 1198 ms. Trace:
I20260620 19:59:01.661564 6232 rpcz_store.cc:276] 0620 19:59:00.449500 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.111531 (+662031us) service_pool.cc:224] Handling call
0620 19:59:01.111963 (+ 432us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:01.647365 (+535402us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.647573 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:01.654944 6344 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.561s (SENT) [suppressed 48 similar messages]
W20260620 19:59:01.315582 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:01.662809 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
W20260620 19:59:01.102224 8455 consensus_meta.cc:287] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.577s user 0.006s sys 0.003s
I20260620 19:59:01.640360 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1498) took 2404 ms. Trace:
I20260620 19:59:01.667389 6374 rpcz_store.cc:276] 0620 19:58:59.236249 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.237089 (+ 840us) service_pool.cc:224] Handling call
0620 19:58:59.329584 (+ 92495us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.974411 (+1644827us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.640086 (+665675us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400341554. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:01.640285 (+ 199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3779517184}
I20260620 19:59:01.197696 8247 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.668783 8528 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:01.657671 8466 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:01.623914 8456 consensus_meta.cc:287] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.673s user 0.009s sys 0.000s
I20260620 19:59:01.669221 8528 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:01.669662 8528 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:01.669911 8466 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:01.159082 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:01.509770 8434 consensus_meta.cc:287] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.594s user 0.009s sys 0.000s
I20260620 19:59:01.664752 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1988) took 1063 ms. Trace:
I20260620 19:59:01.295306 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:01.196607 6903 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
W20260620 19:59:01.675450 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:01.653381 6909 rpcz_store.cc:269] 0620 19:58:59.079363 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.383296 (+303933us) service_pool.cc:224] Handling call
0620 19:59:01.293120 (+1909824us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54004f8dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:01.293309 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2389201664}
W20260620 19:59:01.104593 8398 consensus_meta.cc:287] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.747s user 0.009s sys 0.001s
I20260620 19:59:01.677137 6232 rpcz_store.cc:276] 0620 19:59:00.601112 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.663853 (+1062741us) service_pool.cc:224] Handling call
0620 19:59:01.664205 (+ 352us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.664404 (+ 199us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:01.664440 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:01.664492 (+ 52us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:01.664530 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.664603 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.664701 (+ 98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.208966 8436 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:01.206700 8328 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:01.634164 8469 consensus_meta.cc:287] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.599s user 0.007s sys 0.000s
I20260620 19:59:01.214063 8454 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:01.683426 6911 raft_consensus.cc:1240] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:01.102734 8273 raft_consensus.cc:697] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:01.196478 8474 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:01.685408 6903 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 592) took 1171 ms (client timeout 1558 ms). Trace:
I20260620 19:59:01.685827 6909 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:01.695086 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 765) took 2965 ms. Trace:
I20260620 19:59:01.102741 8467 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.201331 8435 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:01.308952 8461 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:01.686255 8443 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:53406 (request call id 739) took 4310 ms. Trace:
W20260620 19:59:01.643260 6223 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1953) took 2522 ms (client timeout 2188 ms). Trace:
W20260620 19:59:01.698948 6223 rpcz_store.cc:269] 0620 19:58:59.121136 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:59.951518 (+830382us) service_pool.cc:224] Handling call
0620 19:59:01.642971 (+1691453us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b54000b4854. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:01.643162 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":925177728}
W20260620 19:59:01.688865 6230 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.026s sys 0.099s
W20260620 19:59:01.700210 6230 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.026s sys 0.099s
I20260620 19:59:01.306878 8496 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.690341 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 569) took 3959 ms. Trace:
W20260620 19:59:01.692272 6875 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.000s sys 0.013s
I20260620 19:59:01.692881 6232 raft_consensus.cc:1240] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260620 19:59:01.695389 6909 rpcz_store.cc:276] 0620 19:58:58.729301 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.685318 (+2956017us) service_pool.cc:224] Handling call
0620 19:59:01.685656 (+ 338us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.694887 (+ 9231us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.695045 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:01.703380 6875 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.574s user 0.000s sys 0.013s
I20260620 19:59:01.681969 8434 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.703260 6911 rpcz_store.cc:276] 0620 19:58:57.731034 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.682422 (+3951388us) service_pool.cc:224] Handling call
0620 19:59:01.683182 (+ 760us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.690098 (+ 6916us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.690287 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.704337 6909 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:01.705298 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 766) took 2975 ms. Trace:
I20260620 19:59:01.705428 6911 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260620 19:59:01.705814 6911 pending_rounds.cc:85] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting all ops after (but not including) 5
I20260620 19:59:01.658483 8367 consensus_queue.cc:237] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.504534 8410 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.706518 8434 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.696254 6908 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:01.703848 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 574549us
I20260620 19:59:01.101926 8001 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:01.705734 6909 rpcz_store.cc:276] 0620 19:58:58.729553 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.703847 (+2974294us) service_pool.cc:224] Handling call
0620 19:59:01.704138 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.705011 (+ 873us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.705235 (+ 224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.710141 6909 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:01.706279 6911 pending_rounds.cc:107] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260620 19:59:01.710583 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 769) took 2980 ms. Trace:
I20260620 19:59:01.710963 6911 raft_consensus.cc:2889] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:01.711038 6909 rpcz_store.cc:276] 0620 19:58:58.730219 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.709511 (+2979292us) service_pool.cc:224] Handling call
0620 19:59:01.709836 (+ 325us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.710424 (+ 588us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.710545 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.692553 8432 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:01.678836 8455 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.716607 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 767) took 2986 ms. Trace:
I20260620 19:59:01.717221 6909 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 4. Ops: []
W20260620 19:59:01.694675 6903 rpcz_store.cc:269] 0620 19:59:00.513892 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.517832 (+ 3940us) service_pool.cc:224] Handling call
0620 19:59:01.685337 (+1167505us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.718585 6911 rpcz_store.cc:276] 0620 19:58:58.729774 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.704886 (+2975112us) service_pool.cc:224] Handling call
0620 19:59:01.705135 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.716385 (+ 11250us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.716557 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.207082 8516 consensus_queue.cc:237] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.687903 8398 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.720525 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 770) took 2990 ms. Trace:
I20260620 19:59:01.719337 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:01.691052 8469 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.689827 8436 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.721312 6909 rpcz_store.cc:276] 0620 19:58:58.730437 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.711772 (+2981335us) service_pool.cc:224] Handling call
0620 19:59:01.717095 (+ 5323us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.720310 (+ 3215us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.720477 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.714977 8432 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:01.714000 6905 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.005s sys 0.028s
I20260620 19:59:01.727120 6908 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260620 19:59:01.733209 6910 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:01.698773 8443 rpcz_store.cc:276] 0620 19:58:57.383126 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.600304 (+217178us) service_pool.cc:224] Handling call
0620 19:58:58.275840 (+675536us) ts_tablet_manager.cc:934] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
0620 19:59:00.566551 (+2290711us) tablet_copy_client.cc:368] Tablet copy session begun
0620 19:59:01.205666 (+639115us) tablet_metadata.cc:665] Metadata flushed
0620 19:59:01.205787 (+ 121us) tablet_copy_client.cc:497] Wrote new tablet metadata
0620 19:59:01.635162 (+429375us) tablet_copy_client.cc:418] Wrote new consensus metadata
0620 19:59:01.676718 (+ 41556us) tablet_replica.cc:206] Creating consensus instance
0620 19:59:01.686077 (+ 9359us) ts_tablet_manager.cc:989] Replica registered.
0620 19:59:01.686141 (+ 64us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":168398,"tablet-copy.queue_time_us":674289,"thread_start_us":574162,"threads_started":2}
I20260620 19:59:01.726727 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 779) took 2927 ms. Trace:
I20260620 19:59:01.713217 8410 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:01.735931 6911 rpcz_store.cc:276] 0620 19:58:58.792948 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.719867 (+2926919us) service_pool.cc:224] Handling call
0620 19:59:01.720058 (+ 191us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.720221 (+ 163us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:01.720243 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:01.720282 (+ 39us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:01.720300 (+ 18us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.720332 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.720404 (+ 72us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.734846 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 559) took 4047 ms. Trace:
I20260620 19:59:01.737617 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 573) took 2660 ms. Trace:
I20260620 19:59:01.737844 6908 rpcz_store.cc:276] 0620 19:58:57.687157 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.682818 (+3995661us) service_pool.cc:224] Handling call
0620 19:59:01.696103 (+ 13285us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.733356 (+ 37253us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.734793 (+ 1437us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.737994 6911 rpcz_store.cc:276] 0620 19:58:59.077663 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.736864 (+2659201us) service_pool.cc:224] Handling call
0620 19:59:01.737123 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.737291 (+ 168us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:01.737321 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:01.737366 (+ 45us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:01.737402 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.737452 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.737566 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.739687 6908 raft_consensus.cc:1240] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:01.739826 6911 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:01.740373 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 577) took 2661 ms. Trace:
I20260620 19:59:01.740454 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 786) took 2643 ms. Trace:
W20260620 19:59:01.741150 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 78414.4us
I20260620 19:59:01.743873 6271 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:01.744879 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
W20260620 19:59:01.733618 6905 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s user 0.005s sys 0.028s
I20260620 19:59:01.732417 8432 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:01.740787 6908 rpcz_store.cc:276] 0620 19:58:59.078689 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.739232 (+2660543us) service_pool.cc:224] Handling call
0620 19:59:01.739539 (+ 307us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.740171 (+ 632us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.740327 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27648}
I20260620 19:59:01.749554 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1976) took 1299 ms. Trace:
I20260620 19:59:01.740799 6911 rpcz_store.cc:276] 0620 19:58:59.096617 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.739262 (+2642645us) service_pool.cc:224] Handling call
0620 19:59:01.739548 (+ 286us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.740270 (+ 722us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.740414 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15872}
I20260620 19:59:01.751850 6231 rpcz_store.cc:276] 0620 19:59:00.451006 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.655633 (+1204627us) service_pool.cc:224] Handling call
0620 19:59:01.656039 (+ 406us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.656258 (+ 219us) raft_consensus.cc:1534] Early marking committed up to index 4
0620 19:59:01.677890 (+ 21632us) log.cc:844] Serialized 14 byte log entry
0620 19:59:01.749156 (+ 71266us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:01.749223 (+ 67us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:01.749281 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.749361 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.749493 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23328,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":289,"thread_start_us":70405,"threads_started":1}
I20260620 19:59:01.756466 6271 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:01.757572 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:01.753085 8466 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:01.753805 8467 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.754140 8436 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.755472 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 789) took 2262 ms. Trace:
I20260620 19:59:01.737516 8533 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:01.761760 6911 rpcz_store.cc:276] 0620 19:58:59.493173 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.753486 (+2260313us) service_pool.cc:224] Handling call
0620 19:59:01.754873 (+ 1387us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.755074 (+ 201us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:01.755110 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:01.755171 (+ 61us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:01.755218 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.755287 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.755412 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.772787 8538 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:59:01.776504 8538 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:01.776916 8538 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:01.778028 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1968) took 1402 ms. Trace:
I20260620 19:59:01.755383 8443 tablet_copy_client.cc:806] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 0 data blocks...
I20260620 19:59:01.779174 8443 tablet_copy_client.cc:670] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Starting download of 1 WAL segments...
I20260620 19:59:01.778535 6230 rpcz_store.cc:276] 0620 19:59:00.375672 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.018263 (+642591us) service_pool.cc:224] Handling call
0620 19:59:01.043386 (+ 25123us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:01.043774 (+ 388us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:01.043810 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:01.044350 (+ 540us) log.cc:844] Serialized 26 byte log entry
0620 19:59:01.700609 (+656259us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:01.700646 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.700708 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:01.777727 (+ 77019us) raft_consensus.cc:1685] finished
0620 19:59:01.777750 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.777934 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1925,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":22010,"thread_start_us":613966,"threads_started":1,"wal-append.queue_time_us":655491}
I20260620 19:59:01.780716 8436 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:01.781968 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1950) took 2701 ms. Trace:
I20260620 19:59:01.782234 6229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1963) took 1412 ms. Trace:
I20260620 19:59:01.782671 6229 rpcz_store.cc:276] 0620 19:59:00.369353 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.015134 (+645781us) service_pool.cc:224] Handling call
0620 19:59:01.015418 (+ 284us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:01.015737 (+ 319us) raft_consensus.cc:1534] Early marking committed up to index 2
0620 19:59:01.019990 (+ 4253us) log.cc:844] Serialized 14 byte log entry
0620 19:59:01.105956 (+ 85966us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:01.249051 (+143095us) log.cc:844] Serialized 26 byte log entry
0620 19:59:01.249305 (+ 254us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:01.249606 (+ 301us) log.cc:844] Serialized 14 byte log entry
0620 19:59:01.249831 (+ 225us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.249922 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:01.782067 (+532145us) raft_consensus.cc:1685] finished
0620 19:59:01.782084 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.782188 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13597,"prepare.queue_time_us":5140,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":88,"raft.queue_time_us":16401,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":109698,"thread_start_us":71945,"threads_started":1,"wal-append.queue_time_us":689941}
I20260620 19:59:01.786317 8496 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.774958 8533 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.787845 8398 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:01.787289 6903 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:59:01.788612 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 586) took 2284 ms. Trace:
I20260620 19:59:01.755376 8460 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:59:01.782018 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1860) took 1178 ms. Trace:
I20260620 19:59:01.775740 6271 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:01.782524 6224 rpcz_store.cc:276] 0620 19:58:59.081030 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.183762 (+1102732us) service_pool.cc:224] Handling call
0620 19:59:00.310413 (+126651us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.351935 (+ 41522us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.781726 (+1429791us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54000b6650. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:01.781897 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":290557824}
I20260620 19:59:01.791209 6378 rpcz_store.cc:276] 0620 19:59:00.603364 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.820095 (+216731us) service_pool.cc:224] Handling call
0620 19:59:00.820401 (+ 306us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:00.832825 (+ 12424us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.832851 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:01.073116 (+240265us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b2c000a54f8. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b71b3c71 00007fa6b71c35d2 00007fa6b88f5de1 00007fa6b88ba81a 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22
0620 19:59:01.073663 (+ 547us) log.cc:844] Serialized 52 byte log entry
0620 19:59:01.689970 (+616307us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:01.690022 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.690113 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:01.781714 (+ 91601us) raft_consensus.cc:1685] finished
0620 19:59:01.781733 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.781952 (+ 219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12051,"raft.queue_time_us":12479,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"spinlock_wait_cycles":26997120,"thread_start_us":474208,"threads_started":1,"wal-append.queue_time_us":625883}
I20260620 19:59:01.789038 6903 rpcz_store.cc:276] 0620 19:58:59.504043 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.759335 (+2255292us) service_pool.cc:224] Handling call
0620 19:59:01.786800 (+ 27465us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.788334 (+ 1534us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.788546 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:01.799984 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:01.759889 8466 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:01.801339 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:01.797250 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 558) took 4110 ms. Trace:
I20260620 19:59:01.801242 8496 consensus_queue.cc:1059] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 6.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260620 19:59:01.802331 8533 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.802788 8496 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:01.802825 8466 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:01.802325 6905 rpcz_store.cc:276] 0620 19:58:57.686712 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.022791 (+3336079us) service_pool.cc:224] Handling call
0620 19:59:01.023081 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.023296 (+ 215us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:01.104212 (+ 80916us) log.cc:844] Serialized 14 byte log entry
0620 19:59:01.755513 (+651301us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:01.759513 (+ 4000us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:01.762916 (+ 3403us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.773306 (+ 10390us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.795338 (+ 22032us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":230633,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":8045,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":10540,"thread_start_us":712237,"threads_started":2,"wal-append.queue_time_us":655117}
I20260620 19:59:01.806510 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 790) took 2257 ms. Trace:
I20260620 19:59:01.803292 8496 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:01.807343 6905 rpcz_store.cc:276] 0620 19:58:59.548906 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.805593 (+2256687us) service_pool.cc:224] Handling call
0620 19:59:01.805923 (+ 330us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.806127 (+ 204us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:01.806158 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:01.806209 (+ 51us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:01.806249 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.806312 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.806449 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.808717 8496 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.803337 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:01.810057 8496 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:01.809300 6905 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:59:01.732676 8273 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.799044 8436 raft_consensus.cc:697] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:01.839663 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 916443us
W20260620 19:59:01.840811 6344 leader_election.cc:336] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.859s (SENT)
I20260620 19:59:01.840886 8398 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:01.841490 8398 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.819238 8469 leader_election.cc:290] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.848290 8436 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.869264 8455 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.870011 8528 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.870970 8469 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:01.871949 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1840) took 2721 ms. Trace:
I20260620 19:59:01.872062 8528 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.871670 8469 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.872370 6227 rpcz_store.cc:276] 0620 19:58:59.150359 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.280455 (+1130096us) service_pool.cc:224] Handling call
0620 19:59:00.749419 (+468964us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.802460 (+ 53041us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.871756 (+1069296us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b5400372dd0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:01.871879 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":840478592}
I20260620 19:59:01.876124 8469 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:01.879601 6226 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 331) took 1446 ms (client timeout 1624 ms). Trace:
W20260620 19:59:01.880016 6226 rpcz_store.cc:269] 0620 19:59:00.433650 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.596662 (+163012us) service_pool.cc:224] Handling call
0620 19:59:01.879437 (+1282775us) spinlock_profiling.cc:243] Waited 84.1 ms on lock 0x7b54001d42d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:01.879544 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":193427200}
I20260620 19:59:01.884102 8538 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.886520 8538 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:01.887784 6874 leader_election.cc:336] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.652s (SENT)
I20260620 19:59:01.888368 6874 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:01.829260 8542 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:01.849622 8537 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:01.864430 8398 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:01.901008 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 128351us
I20260620 19:59:01.909751 8537 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.913132 6225 raft_consensus.cc:2410] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:01.864184 8466 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.807705 8432 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.854164 8340 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.842890 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 752) took 2049 ms. Trace:
I20260620 19:59:01.904836 6271 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:01.901558 8454 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:01.914441 8537 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.919040 8406 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:01.923853 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 83258.9us
I20260620 19:59:01.924247 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:01.923693 6875 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:01.917492 6307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1725) took 1101 ms. Trace:
I20260620 19:59:01.925844 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:01.964131 6875 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:01.964732 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:01.965493 6875 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:01.965767 8484 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:01.938613 8460 raft_consensus.cc:697] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:01.966542 8484 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:01.962891 8491 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:01.968081 8491 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.967571 6307 rpcz_store.cc:276] 0620 19:59:00.816474 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.818723 (+ 2249us) service_pool.cc:224] Handling call
0620 19:59:00.819024 (+ 301us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:00.965165 (+146141us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.965195 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:00.971966 (+ 6771us) log.cc:844] Serialized 26 byte log entry
0620 19:59:01.116187 (+144221us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:01.116242 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.911401 (+795159us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54003a6954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:01.911467 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:01.911533 (+ 66us) raft_consensus.cc:1685] finished
0620 19:59:01.911564 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.917426 (+ 5862us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":495,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":11678,"spinlock_wait_cycles":335018752,"thread_start_us":103622,"threads_started":1,"wal-append.queue_time_us":651730}
I20260620 19:59:01.969981 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:01.970130 8491 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.926868 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:01.970650 6911 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:01.945989 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:01.972239 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 67496.1us
I20260620 19:59:01.973920 6271 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:01.966586 6875 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:01.975394 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:01.976969 6911 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:01.977634 8406 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:01.980021 8407 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:01.981287 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 756) took 1810 ms. Trace:
I20260620 19:59:01.981745 6903 rpcz_store.cc:276] 0620 19:59:00.170670 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.810741 (+1640071us) service_pool.cc:224] Handling call
0620 19:59:01.811058 (+ 317us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:01.811681 (+ 623us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:01.811711 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:01.812238 (+ 527us) log.cc:844] Serialized 26 byte log entry
0620 19:59:01.979736 (+167498us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:01.979789 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.979876 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:01.981079 (+ 1203us) raft_consensus.cc:1685] finished
0620 19:59:01.981097 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:01.981231 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":84,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":81,"thread_start_us":167105,"threads_started":1,"wal-append.queue_time_us":167427}
I20260620 19:59:01.977185 6875 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:01.984491 8484 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:01.984561 6875 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:01.984762 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:01.985074 8484 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:01.985435 6903 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:01.986300 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:01.986479 8406 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:01.985917 8484 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.986922 8406 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:01.987057 6903 raft_consensus.cc:2393] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:01.987294 8406 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:01.974516 8454 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:01.990794 8454 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.991025 8466 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:01.992861 6911 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:01.946959 6380 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260620 19:59:01.989535 6273 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:01.945940 8398 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:01.967299 8460 consensus_queue.cc:237] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:01.998761 8398 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.000777 8398 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:01.996820 8436 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.005815 6271 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.008401 6271 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:01.969726 6905 rpcz_store.cc:276] 0620 19:58:59.793249 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.808718 (+2015469us) service_pool.cc:224] Handling call
0620 19:59:01.809062 (+ 344us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:01.823242 (+ 14180us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:01.832697 (+ 9455us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:01.975127 8548 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:02.013978 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:02.014489 6904 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 773) took 3223 ms (client timeout 1717 ms). Trace:
I20260620 19:59:02.012380 8466 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:02.017025 8466 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.017400 8466 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:02.017700 8457 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:02.016559 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1866) took 1375 ms. Trace:
I20260620 19:59:02.019382 8432 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:02.025171 6227 raft_consensus.cc:1240] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:02.047091 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.047731 6224 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:02.050314 6905 raft_consensus.cc:1240] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
W20260620 19:59:02.052213 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.023021 8545 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:02.045349 8548 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:02.063218 6273 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:02.016355 6904 rpcz_store.cc:269] 0620 19:58:58.791221 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:58.851917 (+ 60696us) service_pool.cc:224] Handling call
0620 19:59:02.014317 (+3162400us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b5400622fd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:02.014422 (+ 105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3357495552}
I20260620 19:59:02.067417 6273 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:02.068918 8545 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:02.069155 8436 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:02.071813 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 764) took 1314 ms. Trace:
W20260620 19:59:02.062032 8557 log.cc:927] Time spent T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260620 19:59:02.083050 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 758) took 1656 ms. Trace:
I20260620 19:59:02.088290 6911 rpcz_store.cc:276] 0620 19:59:00.426168 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.993495 (+1567327us) service_pool.cc:224] Handling call
0620 19:59:01.993742 (+ 247us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.993908 (+ 166us) raft_consensus.cc:1534] Early marking committed up to index 5
0620 19:59:02.012996 (+ 19088us) log.cc:844] Serialized 14 byte log entry
0620 19:59:02.081145 (+ 68149us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:02.081197 (+ 52us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:02.081240 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:02.082888 (+ 1648us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:02.083002 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":67024,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":75,"thread_start_us":71241,"threads_started":2}
I20260620 19:59:02.091053 6911 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:02.064569 8548 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.099243 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.100374 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.101887 6301 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:02.102380 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.103405 6302 raft_consensus.cc:2393] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:02.101804 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.119156 6911 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260620 19:59:02.119518 6911 pending_rounds.cc:85] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting all ops after (but not including) 4
I20260620 19:59:02.119740 6911 pending_rounds.cc:107] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260620 19:59:02.121027 6380 raft_consensus.cc:1240] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:02.107050 6874 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.119948 6911 raft_consensus.cc:2889] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:02.043646 6379 rpcz_store.cc:276] 0620 19:59:00.640769 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.832108 (+191339us) service_pool.cc:224] Handling call
0620 19:59:00.832364 (+ 256us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.869408 (+ 37044us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:00.869454 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:00.869515 (+ 61us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:00.869564 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:02.016319 (+1146755us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b54000249d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:02.016370 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:02.016500 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84613888}
I20260620 19:59:02.123368 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.140211 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.139694 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:02.154554 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 47611us
I20260620 19:59:02.155438 6304 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:02.158910 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.082504 6905 rpcz_store.cc:276] 0620 19:59:00.757276 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:02.047222 (+1289946us) service_pool.cc:224] Handling call
0620 19:59:02.050078 (+ 2856us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:02.059802 (+ 9724us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:02.071755 (+ 11953us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:02.157511 6301 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:02.125435 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.124403 6298 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:59:02.120541 8436 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:02.158562 8461 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.158053 8398 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:59:02.118572 8548 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:02.156661 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:02.173915 6195 leader_election.cc:336] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.573s (SENT)
I20260620 19:59:02.182973 6300 raft_consensus.cc:2368] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
I20260620 19:59:02.183357 6195 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.184671 6195 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:02.173914 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.179322 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.185796 6306 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:02.186760 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:02.200883 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:02.201766 6874 leader_election.cc:336] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.626s (SENT)
I20260620 19:59:02.202307 6874 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.208663 6874 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.185873 8398 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.185647 8436 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:02.190670 8491 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260620 19:59:02.212085 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.196938 8270 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:02.200479 8273 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:02.225414 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:02.214998 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 206636us
I20260620 19:59:02.205703 8474 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.186179 8461 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.197068 8328 consensus_queue.cc:1059] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260620 19:59:02.212802 8446 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.214469 8398 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:02.189860 8466 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.231169 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.231648 6305 raft_consensus.cc:2376] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5. Re-sending same reply.
I20260620 19:59:02.227447 6271 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.229882 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.230379 6300 raft_consensus.cc:1240] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:02.224901 8270 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.233896 6301 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:02.236032 6306 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:02.234876 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.237330 6305 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:02.236017 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.230762 8328 raft_consensus.cc:3055] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:02.238415 8328 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:02.239297 8328 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.241114 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.241802 6905 raft_consensus.cc:2376] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2. Re-sending same reply.
I20260620 19:59:02.241894 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 749) took 2449 ms. Trace:
I20260620 19:59:02.242314 6906 rpcz_store.cc:276] 0620 19:58:59.792293 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.316062 (+523769us) service_pool.cc:224] Handling call
0620 19:59:02.241672 (+1925610us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54005eef54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:02.241833 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":375058304}
I20260620 19:59:02.243130 8432 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:02.243498 6909 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260620 19:59:02.243844 6909 pending_rounds.cc:85] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting all ops after (but not including) 5
I20260620 19:59:02.244045 6909 pending_rounds.cc:107] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260620 19:59:02.244254 6909 raft_consensus.cc:2889] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:02.244572 8466 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.240460 8328 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:02.245909 8001 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.245975 8466 consensus_queue.cc:1059] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:02.247364 8432 raft_consensus.cc:3055] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 19:59:02.247707 8432 raft_consensus.cc:740] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:02.248483 8432 consensus_queue.cc:260] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.227200 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 757) took 1946 ms. Trace:
I20260620 19:59:02.251227 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.251340 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.252179 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.254189 8432 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:59:02.258566 6227 raft_consensus.cc:1240] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:02.260608 8407 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.260870 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.261075 8407 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.262295 8461 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.235356 8538 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:02.262493 6902 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 574) took 3184 ms (client timeout 2643 ms). Trace:
I20260620 19:59:02.262948 6227 raft_consensus.cc:1240] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:02.272488 6911 raft_consensus.cc:1240] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 1. Current term is 2. Ops: []
I20260620 19:59:02.263445 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1861) took 1641 ms. Trace:
I20260620 19:59:02.263962 6908 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:02.263720 8406 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.264328 6905 raft_consensus.cc:1240] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:02.237454 8270 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:02.273867 6195 leader_election.cc:336] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.671s (SENT)
I20260620 19:59:02.273844 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.319756 6195 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.319944 6227 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:59:02.250010 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.321554 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.273757 6344 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:02.298934 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:02.295732 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.300731 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.323618 6911 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:02.302948 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.300122 8327 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.326103 6223 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: [4.6-5.7]
I20260620 19:59:02.303546 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.304111 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.326040 8327 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.330868 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.334110 8327 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:02.271039 6902 rpcz_store.cc:269] 0620 19:58:59.077953 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.220455 (+1142502us) service_pool.cc:224] Handling call
0620 19:59:02.262270 (+2041815us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000deb54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:02.262424 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434141440}
I20260620 19:59:02.335335 8454 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.335649 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.295464 8436 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:02.305581 8536 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.470s user 0.003s sys 0.000s
I20260620 19:59:02.319160 6305 raft_consensus.cc:2410] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260620 19:59:02.320693 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 46842us
I20260620 19:59:02.324460 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:02.340747 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.341241 6911 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
W20260620 19:59:02.341923 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 68209.3us
I20260620 19:59:02.342100 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.343222 6344 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:02.340369 6874 leader_election.cc:336] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.753s (SENT)
I20260620 19:59:02.345471 6874 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.346730 8541 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:02.347188 8541 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:02.347936 8541 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:02.315646 8484 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.350947 8484 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:02.352402 8565 log.cc:927] Time spent T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260620 19:59:02.353188 8484 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.353420 8327 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.353912 8327 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:02.355973 6904 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:59:02.356669 8537 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:02.274689 8538 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:02.359326 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:02.360247 8538 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:59:02.360973 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.361179 8460 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:02.361724 6905 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:02.252866 6903 rpcz_store.cc:276] 0620 19:59:00.279492 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.987983 (+1708491us) service_pool.cc:224] Handling call
0620 19:59:01.996819 (+ 8836us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:01.997025 (+ 206us) raft_consensus.cc:1534] Early marking committed up to index 7
0620 19:59:02.012185 (+ 15160us) log.cc:844] Serialized 14 byte log entry
0620 19:59:02.207673 (+195488us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:02.211678 (+ 4005us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:02.213531 (+ 1853us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:02.222912 (+ 9381us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:02.226052 (+ 3140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":516,"prepare.queue_time_us":65556,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":23,"raft.queue_time_us":19141,"raft.run_wall_time_us":4475,"spinlock_wait_cycles":3552512,"thread_start_us":155701,"threads_started":2,"wal-append.queue_time_us":189908}
I20260620 19:59:02.363824 6903 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260620 19:59:02.364137 6903 pending_rounds.cc:85] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting all ops after (but not including) 6
I20260620 19:59:02.364372 6903 pending_rounds.cc:107] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260620 19:59:02.364630 6903 raft_consensus.cc:2889] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:02.274885 6373 rpcz_store.cc:276] 0620 19:59:00.621835 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.822231 (+200396us) service_pool.cc:224] Handling call
0620 19:59:00.822517 (+ 286us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:00.981149 (+158632us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:02.263219 (+1282070us) spinlock_profiling.cc:243] Waited 91.3 ms on lock 0x7b54001216d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:02.263383 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210004736}
I20260620 19:59:02.374511 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 781) took 3559 ms. Trace:
I20260620 19:59:02.374948 6909 rpcz_store.cc:276] 0620 19:58:58.815049 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.728251 (+2913202us) service_pool.cc:224] Handling call
0620 19:59:02.242501 (+514250us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:02.244806 (+ 2305us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:02.374343 (+129537us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54005eef50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:02.374459 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1166727040}
I20260620 19:59:02.304111 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:02.323189 8434 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:02.393368 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.390117 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:02.390336 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 49986.2us
I20260620 19:59:02.400918 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.401664 6380 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:02.456730 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:02.457347 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.458148 6306 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:59:02.365974 6903 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:59:02.475945 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:02.477459 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.478098 6911 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:02.470309 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:02.483068 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 180686us
I20260620 19:59:02.377041 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.471938 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.484072 6909 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:02.484628 6345 connection.cc:256] RPC call timeout handler was delayed by -1.1572385240000358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20260620 19:59:02.485086 6345 connection.cc:256] RPC call timeout handler was delayed by -1.1572385240000358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
I20260620 19:59:02.488986 6231 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:02.409099 8484 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:02.458169 8568 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:02.492872 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:02.496444 6345 connection.cc:256] RPC call timeout handler was delayed by -1.0213385240000434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20099999999999998s for a response
I20260620 19:59:02.496570 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.497699 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.498405 6301 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:02.498991 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.499408 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.495777 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.495821 6305 raft_consensus.cc:1275] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260620 19:59:02.500123 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.501103 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.489123 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.503280 6373 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:02.504206 6302 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:02.487351 8454 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.505877 8454 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.490787 8432 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.506712 8568 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.507171 8432 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.507952 8454 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.508875 8568 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.508977 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.509160 8432 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.510003 8560 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.510659 8560 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.494580 8566 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.512123 8566 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.512707 8560 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.514102 8566 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.514225 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.515271 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.515372 8146 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.515870 8146 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.516284 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.517306 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.517690 6303 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:02.510047 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.520761 8538 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:02.522130 8538 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:02.392680 8458 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260620 19:59:02.525970 6874 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:02.522153 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:02.527060 8454 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:02.527302 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.527565 8446 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:02.527974 8446 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.528314 8446 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.528393 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:02.529021 6906 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.531354 6874 leader_election.cc:400] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
I20260620 19:59:02.531149 8566 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:02.532110 6874 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:02.485045 8542 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.562000 6909 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:02.565021 6271 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:02.563881 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:02.566898 6909 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:02.567008 6911 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:02.568249 6911 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:02.555019 8496 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.575608 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.581920 8496 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.572350 8474 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:02.582933 8474 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.583252 8547 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:02.583678 8547 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.584060 8547 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.584218 8496 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:02.590942 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.564989 8434 consensus_queue.cc:1059] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260620 19:59:02.595160 8406 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.595605 8434 raft_consensus.cc:3055] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:02.596038 8434 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:02.596838 8434 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.597332 8407 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.598053 8434 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.598899 8432 consensus_queue.cc:1059] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:02.600428 8270 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.600809 8436 raft_consensus.cc:3055] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:02.601361 8436 raft_consensus.cc:740] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:02.602789 6228 raft_consensus.cc:1240] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:02.603938 8407 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.602413 8436 consensus_queue.cc:260] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.609726 6906 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:02.615584 8560 consensus_queue.cc:1059] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:02.617096 8528 raft_consensus.cc:3055] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:02.617566 8528 raft_consensus.cc:740] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:02.618726 8528 consensus_queue.cc:260] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.620546 8528 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.622079 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 788) took 3131 ms. Trace:
I20260620 19:59:02.622467 6908 rpcz_store.cc:276] 0620 19:58:59.492806 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.756116 (+2263310us) service_pool.cc:224] Handling call
0620 19:59:01.968891 (+212775us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:02.263795 (+294904us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:02.307506 (+ 43711us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:02.621859 (+314353us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54000deb50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:02.622022 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":715590912}
I20260620 19:59:02.623317 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:02.624332 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 44553.5us
I20260620 19:59:02.583293 8474 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.630045 6874 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:02.630066 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:02.632242 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 46933.7us
I20260620 19:59:02.572125 8273 consensus_queue.cc:1059] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260620 19:59:02.574330 8461 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260620 19:59:02.635426 8484 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.574992 8548 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.637319 6344 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:02.637563 8466 consensus_queue.cc:1059] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:02.638351 8466 consensus_queue.cc:1243] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer 20bd146dc0cb4917a1a5ce05ec609ce1 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260620 19:59:02.638677 8480 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:02.639268 8480 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.639859 8480 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:02.640872 6909 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260620 19:59:02.631685 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 58483us
I20260620 19:59:02.643374 8146 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:02.643956 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.646085 6379 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:02.647867 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:02.610530 8541 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.572563 8542 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:02.656828 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.656562 6345 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.660254 6345 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.658455 6344 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:02.638866 8466 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:02.594224 8496 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260620 19:59:02.662647 6195 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:02.664494 6345 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:02.666066 6195 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.669029 6345 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:02.669904 6909 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:59:02.670485 6302 raft_consensus.cc:1240] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:02.671038 8457 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:02.671818 8542 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.671696 8479 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:02.672820 8496 raft_consensus.cc:3055] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:02.660784 8480 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.673365 8496 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:02.667047 8458 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.664256 8466 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:02.674477 8496 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.675390 8466 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.676159 8496 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:02.676484 8466 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:02.636619 8270 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.673174 8398 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.678790 6345 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.681771 6381 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:02.682886 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.683008 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.685669 6195 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.637002 8560 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260620 19:59:02.662889 8573 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.687776 6345 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:02.689749 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.689285 6195 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.692175 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:02.677284 8273 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:02.692633 6195 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.692812 6909 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:02.657742 8146 consensus_queue.cc:1048] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260620 19:59:02.664508 8541 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:02.663619 8576 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:02.684348 8434 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.685959 8534 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:02.680069 8484 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.698736 8328 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.699532 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.702211 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.695137 8480 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:02.703058 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.750747 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.752103 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.703740 8548 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.704380 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:02.711437 6902 raft_consensus.cc:2393] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:02.712854 8560 consensus_queue.cc:1059] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:02.714685 8537 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:02.693173 8458 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:02.689814 8533 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:02.686568 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.719579 6907 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:59:02.693635 8465 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:02.720196 6225 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 1987) took 2119 ms (client timeout 2223 ms). Trace:
W20260620 19:59:02.785612 6225 rpcz_store.cc:269] 0620 19:59:00.600870 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:01.005867 (+404997us) service_pool.cc:224] Handling call
0620 19:59:02.719996 (+1714129us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b54001e46d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:02.720144 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2083400960}
I20260620 19:59:02.786736 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.723559 6196 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:02.788008 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 64499.7us
I20260620 19:59:02.789677 6196 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:02.694235 8270 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260620 19:59:02.686089 8457 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260620 19:59:02.736728 6910 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 746) took 2946 ms (client timeout 1706 ms). Trace:
I20260620 19:59:02.794116 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:02.794363 6910 rpcz_store.cc:269] 0620 19:58:59.790568 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:00.191298 (+400730us) service_pool.cc:224] Handling call
0620 19:59:02.736493 (+2545195us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b540054fbd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:02.736650 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3542643968}
I20260620 19:59:02.795109 6226 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:02.737299 8273 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:02.738068 8327 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:02.738448 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.748934 6380 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:02.796452 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.797096 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:02.798484 6380 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260620 19:59:02.764760 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:02.764855 6271 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.738107 8469 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:02.771477 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 81772.6us
I20260620 19:59:02.803243 6380 raft_consensus.cc:1240] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:02.803603 6230 raft_consensus.cc:2393] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 2: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:02.804315 6345 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.743496 8534 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.778002 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.779162 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.779798 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.780417 6909 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:46646
I20260620 19:59:02.785001 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.787284 6225 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:02.792038 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.744025 8446 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.807157 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.814843 8446 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:02.794173 8328 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:02.816146 6227 raft_consensus.cc:2393] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:02.794063 8457 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:02.817094 8457 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:02.820842 6223 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:59:02.823215 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.824065 6232 raft_consensus.cc:2410] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:02.807976 6345 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:02.809569 6228 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:02.780884 8458 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:02.808517 6909 raft_consensus.cc:606] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.826815 6909 raft_consensus.cc:614] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:02.748759 8484 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.831213 6271 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:02.828979 8446 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.822729 8534 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:02.831789 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:02.811676 8528 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:02.830214 8327 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:02.840420 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.830348 8458 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:02.831789 8480 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:02.807057 8583 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260620 19:59:02.875027 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:02.799968 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 111579us
I20260620 19:59:02.833540 6905 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:02.898723 6380 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:59:02.869913 8407 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.885589 8548 raft_consensus.cc:3055] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:02.839176 8582 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:02.896675 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:02.897352 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.879081 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.826349 8560 consensus_queue.cc:1059] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260620 19:59:02.823210 8469 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.830106 8270 consensus_queue.cc:1059] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
W20260620 19:59:02.901859 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 110468us
I20260620 19:59:02.831642 8345 consensus_queue.cc:1059] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
W20260620 19:59:02.905355 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 65586.4us
I20260620 19:59:02.936884 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:02.907068 6231 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:02.945096 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:02.945719 6196 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:02.894534 8580 consensus_queue.cc:1059] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260620 19:59:02.894062 8460 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:02.875456 8446 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.960381 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:02.961968 6230 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
W20260620 19:59:02.964115 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:02.907676 8583 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:02.878813 8474 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.905028 8581 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:59:02.969298 6196 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.989723 6301 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260620 19:59:03.003743 6301 pending_rounds.cc:85] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 6
I20260620 19:59:03.004086 6301 pending_rounds.cc:107] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260620 19:59:03.004351 6301 raft_consensus.cc:2889] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:02.969705 6198 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:03.013731 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 44498.9us
I20260620 19:59:03.014814 6196 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:02.951184 8560 consensus_queue.cc:1243] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260620 19:59:02.959681 6231 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
W20260620 19:59:03.015748 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 46107.6us
I20260620 19:59:03.012144 8432 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:02.966578 8460 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:03.019533 8328 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:02.997529 8582 consensus_queue.cc:1059] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260620 19:59:03.020682 8469 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:03.021052 8273 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:03.024736 6228 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:02.989897 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:02.970180 8589 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260620 19:59:03.025372 8457 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:02.904629 8548 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:02.975045 8584 log.cc:927] Time spent T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260620 19:59:03.020705 6196 leader_election.cc:304] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:03.033339 6198 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:03.013847 8581 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.907253 8145 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:02.907804 8407 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:02.952519 8398 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:02.988385 8541 raft_consensus.cc:3055] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:03.040256 8273 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:02.969950 8491 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:59:02.987768 8484 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:03.039315 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 207932us
W20260620 19:59:03.043251 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 98156.4us
W20260620 19:59:03.044183 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 107336us
I20260620 19:59:03.018478 8270 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:02.981366 8547 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.038544 8328 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:03.037078 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 297286us
I20260620 19:59:03.054066 6374 raft_consensus.cc:1240] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:03.046499 6906 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:03.013080 8583 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.088474 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:03.055491 8273 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:03.092041 6375 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:03.101399 8534 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.052569 8465 raft_consensus.cc:2764] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260620 19:59:03.102270 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.103247 8145 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:03.104794 8407 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:03.096213 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:03.136240 6273 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:03.137257 6273 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:03.105314 6271 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:03.106779 8434 consensus_queue.cc:1059] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:03.109303 8273 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.104809 8436 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:03.109799 8533 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:03.110188 8432 raft_consensus.cc:3055] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:03.110553 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 766) took 2227 ms. Trace:
I20260620 19:59:03.111828 8446 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:03.112044 8466 raft_consensus.cc:3055] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:03.113811 8583 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:03.120831 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 67737.4us
I20260620 19:59:03.052701 8581 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:03.086277 8270 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:03.157102 8597 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:03.161041 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:03.161602 6304 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:03.163983 8273 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:03.147866 8582 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:03.166500 8582 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:03.166862 8582 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.167863 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:03.168582 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 63334.8us
I20260620 19:59:03.158088 6904 rpcz_store.cc:276] 0620 19:59:00.883482 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:02.081857 (+1198375us) service_pool.cc:224] Handling call
0620 19:59:02.082157 (+ 300us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:02.356404 (+274247us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:03.095013 (+738609us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400740254. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:03.110484 (+ 15471us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628974336}
I20260620 19:59:03.144500 6379 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:03.098744 8599 consensus_queue.cc:1059] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.064s
I20260620 19:59:03.159394 8432 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:03.173394 6344 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:03.173839 8432 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.174662 8573 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:03.095844 8328 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.142743 6345 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:03.052650 8541 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:03.052421 8327 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:03.153301 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:03.153963 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.199470 6302 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:59:03.202057 6271 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:03.160471 8270 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:03.129657 6306 raft_consensus.cc:2393] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:03.181905 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:03.218796 6228 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:03.148275 8596 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:03.185788 6225 raft_consensus.cc:1240] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:03.186429 6198 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:03.215298 6305 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:03.222863 8599 consensus_queue.cc:1243] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 log is divergent from this leader: its last log entry 9.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260620 19:59:03.216861 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.235714 6303 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:03.200443 6304 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:03.182526 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:03.237831 6231 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:03.220155 6230 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260620 19:59:03.239007 6230 pending_rounds.cc:85] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 5
I20260620 19:59:03.239329 6230 pending_rounds.cc:107] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260620 19:59:03.239612 6230 raft_consensus.cc:2889] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:03.128120 8496 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20260620 19:59:03.223068 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.116067 8457 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:03.248457 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:03.179988 8574 raft_consensus.cc:3055] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:03.156347 8480 raft_consensus.cc:697] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:03.248100 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:03.225955 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 83262.2us
I20260620 19:59:03.256965 6231 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
W20260620 19:59:03.263690 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 77352.7us
I20260620 19:59:03.220239 8580 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:59:03.157251 8566 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260620 19:59:03.159621 8466 raft_consensus.cc:740] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:03.148062 8542 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260620 19:59:03.267549 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.277781 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.280709 6345 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:03.234589 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:03.130132 8548 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.209196 8560 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260620 19:59:03.293376 8474 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:03.283849 8595 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260620 19:59:03.293458 6224 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260620 19:59:03.365103 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 148496us
I20260620 19:59:03.261790 8446 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:03.394997 8598 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:03.263123 8469 consensus_queue.cc:1059] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.145s
I20260620 19:59:03.261977 8574 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:03.247375 8491 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:03.392958 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 85325.4us
I20260620 19:59:03.393754 6305 raft_consensus.cc:2393] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:03.246929 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.247071 8533 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:03.265502 8465 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.202s
I20260620 19:59:03.439082 6223 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260620 19:59:03.401830 8458 consensus_meta.cc:287] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.503s user 0.012s sys 0.008s
I20260620 19:59:03.495079 8458 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.497511 8458 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:03.498203 8597 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:03.402431 8534 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:03.441864 8582 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.427024 8145 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20260620 19:59:03.500535 8582 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:03.402477 6379 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:03.403203 8598 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:03.455681 6344 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:03.472324 6195 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:03.520931 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 48665.9us
I20260620 19:59:03.521701 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:03.521833 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 66203.2us
I20260620 19:59:03.399664 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:03.405500 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:03.463174 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:03.522954 6304 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:03.523115 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 123561us
I20260620 19:59:03.477321 6271 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:03.416862 8496 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:03.513702 8576 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:03.438941 8541 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.467377 8603 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:03.262737 8589 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:03.399628 8548 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:03.398146 8466 consensus_queue.cc:260] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.480484 8560 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:03.523787 6345 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:03.540552 6304 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
W20260620 19:59:03.541559 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.541505 6345 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:03.524163 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 46917.4us
I20260620 19:59:03.460340 8588 raft_consensus.cc:3055] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:03.524767 6195 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:03.396297 8573 consensus_queue.cc:1059] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.120s
I20260620 19:59:03.401386 8446 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.417740 8491 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:03.529640 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.527773 8576 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.528002 8603 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:03.535701 8589 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:03.399942 8599 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:03.478185 8533 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:03.450984 8480 consensus_queue.cc:237] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.526787 8597 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:03.609608 8597 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:03.610000 8597 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:03.610134 8576 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:03.610956 8589 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.612551 8589 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:03.614643 8599 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.616837 8599 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:03.621107 8581 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:03.621590 6128 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47082 (request call id 407) took 11856 ms (client timeout 15000 ms). Trace:
I20260620 19:59:03.622219 6300 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260620 19:59:03.598671 8603 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.623814 6130 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 4 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:03.630867 8598 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.633013 8581 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:03.633587 8581 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:03.634457 6229 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:03.592027 8474 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:03.635499 8491 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:03.593833 8545 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260620 19:59:03.640985 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:03.642294 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:03.643476 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:03.596941 8460 consensus_meta.cc:287] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.554s user 0.004s sys 0.004s
I20260620 19:59:03.646906 8460 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:03.648223 6298 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:03.648602 8538 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:03.649147 8460 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:59:03.591933 8580 consensus_queue.cc:1059] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260620 19:59:03.653051 8580 raft_consensus.cc:3055] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:03.653391 8580 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:03.556154 8583 consensus_queue.cc:1059] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20260620 19:59:03.656736 6307 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:59:03.657197 8270 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.659535 8270 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:03.661337 8328 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.556090 8457 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:03.556142 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.528219 8458 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:03.417462 8574 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.607892 8533 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:03.527983 8479 consensus_queue.cc:1059] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.068s
I20260620 19:59:03.603770 8327 raft_consensus.cc:697] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:03.635188 8474 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.557791 6345 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:03.555830 8452 consensus_queue.cc:1059] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260620 19:59:03.675932 6379 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:03.526669 8534 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:03.669077 6271 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:03.634042 8581 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:03.669617 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.654090 8580 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.676901 6375 raft_consensus.cc:1240] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:03.592481 8560 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.626457 6130 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:03.527482 8496 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:03.690232 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 530029us
I20260620 19:59:03.551798 8588 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:03.693238 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.682940 8491 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader election won for term 2
W20260620 19:59:03.693557 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 135799us
I20260620 19:59:03.556697 8001 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:03.690151 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.698987 6130 catalog_manager.cc:5696] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 3, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 3 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:03.711462 6376 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:03.700783 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:03.712072 6130 catalog_manager.cc:5696] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:03.703457 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:03.713733 6224 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.668191 8458 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:03.679450 8574 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.708429 6230 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:03.708555 6345 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:03.717867 6345 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:03.695562 8583 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:03.709833 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:03.722182 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 197405us
W20260620 19:59:03.621989 6128 rpcz_store.cc:269] 0620 19:58:51.764736 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:51.764917 (+ 181us) service_pool.cc:224] Handling call
0620 19:58:52.953347 (+1188430us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54000ad054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8824470 00007fa6b88c331a 00007fa6be678aac 00007fa6be7770ae 00007fa6bcc3c4a4 00007fa6bcc3c432 00007fa6bcc3c361 00007fa6bcc3c2dc 00007fa6bcc3b592 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:03.621527 (+10668180us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:58:59.104223 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:58:59.107495 (+ 3272us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:58:59.107528 (+ 33us) write_op.cc:435] Acquired schema lock
0620 19:58:59.107546 (+ 18us) tablet.cc:662] Decoding operations
0620 19:58:59.108291 (+ 745us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:58:59.108348 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0620 19:58:59.108373 (+ 25us) tablet.cc:685] Acquiring locks for 6 operations
0620 19:58:59.108857 (+ 484us) tablet.cc:701] Row locks acquired
0620 19:58:59.108874 (+ 17us) write_op.cc:260] PREPARE: finished
0620 19:58:59.109008 (+ 134us) write_op.cc:270] Start()
0620 19:58:59.109109 (+ 101us) write_op.cc:276] Timestamp: P: 1781985539108964 usec, L: 0
0620 19:58:59.109140 (+ 31us) op_driver.cc:348] REPLICATION: starting
0620 19:58:59.142160 (+ 33020us) log.cc:844] Serialized 2457 byte log entry
0620 19:59:00.565819 (+1423659us) op_driver.cc:464] REPLICATION: finished
0620 19:59:01.017703 (+451884us) write_op.cc:301] APPLY: starting
0620 19:59:01.036101 (+ 18398us) tablet.cc:1367] starting BulkCheckPresence
0620 19:59:01.062969 (+ 26868us) tablet.cc:1370] finished BulkCheckPresence
0620 19:59:01.084876 (+ 21907us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:59:01.197918 (+113042us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:59:01.198162 (+ 244us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0620 19:59:01.198215 (+ 53us) write_op.cc:312] APPLY: finished
0620 19:59:01.198504 (+ 289us) log.cc:844] Serialized 53 byte log entry
0620 19:59:01.198710 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:01.198823 (+ 113us) write_op.cc:454] Released schema lock
0620 19:59:01.198872 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0620 19:59:01.199251 (+ 379us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:01.199280 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1302693,"reported_tablets":9,"spinlock_wait_cycles":301468544,"child_traces":[["op",{"apply.queue_time_us":442158,"apply.run_cpu_time_us":3453,"apply.run_wall_time_us":187361,"num_ops":6,"prepare.queue_time_us":316159,"prepare.run_cpu_time_us":24088,"prepare.run_wall_time_us":447551,"raft.queue_time_us":191960,"raft.run_cpu_time_us":58239,"raft.run_wall_time_us":417985,"replication_time_us":1406648,"thread_start_us":1049662,"threads_started":4,"wal-append.queue_time_us":913561,"wal-append.run_cpu_time_us":20229,"wal-append.run_wall_time_us":2132222}]]}
I20260620 19:59:03.722793 6226 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:03.714033 6130 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:03.698987 8270 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:03.724097 6130 catalog_manager.cc:5696] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:03.715273 8001 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:03.727802 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.695201 8613 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:03.729036 6345 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:03.706431 8516 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:03.729596 8452 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:03.718397 8458 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.695488 8597 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.741736 6379 raft_consensus.cc:1240] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:03.750694 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.745944 6224 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:03.742568 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:03.740409 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 44819.3us
I20260620 19:59:03.747365 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:03.750353 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.751963 6375 raft_consensus.cc:2368] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5.
I20260620 19:59:03.752233 6376 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:03.753181 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.753427 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.753800 6377 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:03.746774 8583 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:03.733961 8001 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.754462 8516 raft_consensus.cc:697] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:03.755606 8581 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:03.755342 8516 consensus_queue.cc:237] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.755955 8270 raft_consensus.cc:697] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:03.756856 8270 consensus_queue.cc:237] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.758996 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.759105 8618 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:03.759435 8446 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:03.759980 8328 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:03.759706 8618 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.708958 8460 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.029s
I20260620 19:59:03.727568 8606 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
W20260620 19:59:03.761375 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 57218.3us
I20260620 19:59:03.764914 8588 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.734524 8534 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260620 19:59:03.765928 8588 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.765801 8599 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:03.768200 8599 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:03.727533 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.747273 8573 consensus_queue.cc:1059] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.015s
I20260620 19:59:03.729444 8580 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:03.770845 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.759634 6299 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:03.737968 8452 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:03.780061 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:03.779786 6196 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:03.810931 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.750120 8601 consensus_queue.cc:1059] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260620 19:59:03.814932 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:03.815553 8573 raft_consensus.cc:3055] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:03.750500 8491 raft_consensus.cc:697] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:03.843125 8597 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:03.843997 8601 raft_consensus.cc:3055] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:03.844388 8432 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:03.816383 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.846921 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:03.847642 6228 raft_consensus.cc:2393] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:03.812040 8533 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.820911 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.821964 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.857250 6375 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:03.857417 6381 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:03.858062 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:03.858165 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:03.822129 8496 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:03.727074 8327 consensus_queue.cc:237] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:03.828043 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.831792 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:03.871876 6380 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:03.872267 6379 raft_consensus.cc:2393] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:03.832217 8579 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260620 19:59:03.779062 8452 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:03.834884 6231 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:03.836794 6226 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
W20260620 19:59:03.837858 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.876052 6231 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:03.877583 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:03.840654 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 60961.5us
I20260620 19:59:03.772904 8603 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:03.842010 6299 raft_consensus.cc:2393] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:03.842250 8573 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:03.844451 8601 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:03.830394 8474 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:03.851608 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:03.841513 8583 raft_consensus.cc:697] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:03.992043 6874 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:03.910646 6908 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:03.909643 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:03.941826 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.914541 6301 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 587) took 1236 ms. Trace:
I20260620 19:59:04.083653 6306 raft_consensus.cc:2393] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:04.083757 6301 rpcz_store.cc:276] 0620 19:59:02.678149 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:02.678311 (+ 162us) service_pool.cc:224] Handling call
0620 19:59:02.678556 (+ 245us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:03.004729 (+326173us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:03.914325 (+909596us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b54001a9d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:03.914485 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":714639232}
I20260620 19:59:03.931973 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:04.086091 6306 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:59:04.086156 6381 raft_consensus.cc:2368] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5.
I20260620 19:59:03.884059 6232 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260620 19:59:03.914011 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1877) took 1224 ms. Trace:
I20260620 19:59:03.996454 6376 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:04.047806 6302 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:03.941740 6911 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260620 19:59:04.087345 6374 rpcz_store.cc:276] 0620 19:59:02.689818 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:02.689936 (+ 118us) service_pool.cc:224] Handling call
0620 19:59:02.698082 (+ 8146us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:03.090342 (+392260us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:03.913779 (+823437us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400341a54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:03.913951 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818039040}
I20260620 19:59:03.977833 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:04.087158 6232 pending_rounds.cc:85] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 5
I20260620 19:59:04.090818 6232 pending_rounds.cc:107] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260620 19:59:04.091112 6232 raft_consensus.cc:2889] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:04.092347 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1867) took 1198 ms. Trace:
I20260620 19:59:04.092824 6232 rpcz_store.cc:276] 0620 19:59:02.894032 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.860159 (+966127us) service_pool.cc:224] Handling call
0620 19:59:03.883589 (+ 23430us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:04.092041 (+208452us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:04.092281 (+ 240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:03.870548 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:04.087106 8595 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:04.046300 8446 consensus_queue.cc:1059] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260620 19:59:04.005712 6910 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
W20260620 19:59:04.102494 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 98409.1us
I20260620 19:59:03.949056 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:03.964488 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:04.109592 6227 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:04.081079 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:04.110126 6303 raft_consensus.cc:1240] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260620 19:59:04.082947 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:04.110514 6375 raft_consensus.cc:2376] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4. Re-sending same reply.
I20260620 19:59:04.059576 6195 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:03.878254 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:03.906952 6231 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:04.112090 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 52633.7us
I20260620 19:59:04.112457 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:04.110150 6227 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1869) took 1203 ms (client timeout 1581 ms). Trace:
W20260620 19:59:04.082798 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 90794.7us
W20260620 19:59:04.113291 6227 rpcz_store.cc:269] 0620 19:59:02.907570 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.964181 (+1056611us) service_pool.cc:224] Handling call
0620 19:59:04.110083 (+145902us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:04.113798 6304 raft_consensus.cc:1240] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:03.855969 8622 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:04.110818 6226 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:04.111366 6303 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:04.112682 6231 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:04.115485 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:03.964614 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:04.101356 6902 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:04.118260 6232 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:04.119300 6228 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 8.
I20260620 19:59:03.944458 8496 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.124166 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:04.005363 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:04.122256 6303 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:04.063244 8606 raft_consensus.cc:697] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:04.131098 6303 raft_consensus.cc:1240] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:03.911427 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:04.105629 8407 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:04.127549 6304 raft_consensus.cc:1240] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:04.130649 8606 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.135856 6372 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:03.915213 8537 raft_consensus.cc:3055] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
W20260620 19:59:04.137562 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.137701 8537 raft_consensus.cc:740] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:04.142189 6231 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 10. (term mismatch)
I20260620 19:59:04.147464 6231 pending_rounds.cc:85] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 9
I20260620 19:59:04.147763 6231 pending_rounds.cc:107] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 8.10
I20260620 19:59:04.148032 6231 raft_consensus.cc:2889] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:04.149189 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:04.152796 6231 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
W20260620 19:59:04.157966 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:03.911049 8560 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:04.101258 8595 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:04.172230 6226 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:04.187546 8595 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:04.188333 6226 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260620 19:59:04.177492 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.122885 8599 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:04.057437 8001 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.183772 8443 tablet_copy_client.cc:538] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260620 19:59:04.207751 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 88853.5us
W20260620 19:59:04.201532 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 81754.9us
I20260620 19:59:04.228492 6909 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260620 19:59:04.229169 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 814) took 1046 ms. Trace:
I20260620 19:59:04.229472 6909 rpcz_store.cc:276] 0620 19:59:03.182577 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.182730 (+ 153us) service_pool.cc:224] Handling call
0620 19:59:04.228100 (+1045370us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:04.228280 (+ 180us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:04.228931 (+ 651us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:04.229115 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28165888}
I20260620 19:59:04.233533 8547 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:04.063377 8491 consensus_queue.cc:237] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.128772 8458 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:04.194686 8630 consensus_queue.cc:1059] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.128s
I20260620 19:59:04.145267 8496 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.239091 8595 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.333778 8568 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.191847 8575 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.147s
I20260620 19:59:04.184155 8502 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.095s
I20260620 19:59:04.191468 8457 consensus_queue.cc:1059] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.096s
I20260620 19:59:04.225129 6877 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:04.183887 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 776) took 1697 ms. Trace:
W20260620 19:59:04.286803 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 69350.1us
W20260620 19:59:04.393477 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.199847 8599 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:04.316277 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.202332 6905 raft_consensus.cc:2393] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:04.331991 8623 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.387s user 0.001s sys 0.000s
I20260620 19:59:04.398455 8603 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:04.400090 8458 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:04.300005 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.237069 8407 raft_consensus.cc:697] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:04.401422 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:04.430630 6303 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:04.329438 8629 log.cc:927] Time spent T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260620 19:59:04.202229 8479 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:04.198397 8583 consensus_queue.cc:237] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.195580 8601 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.342258 8635 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260620 19:59:04.331418 8633 log.cc:927] Time spent T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260620 19:59:04.279567 8604 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.429970 8639 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.417826 8599 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:04.441459 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:04.201256 8573 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.449401 6374 raft_consensus.cc:1240] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:04.449553 6376 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:04.452878 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:04.392760 8568 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.453466 6374 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:04.430763 8566 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.438009 8581 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.464954 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:04.440276 8432 consensus_meta.cc:287] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.594s user 0.013s sys 0.000s
I20260620 19:59:04.350394 8595 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.453454 8601 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:04.487180 8648 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:04.487957 8648 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.392910 6906 rpcz_store.cc:276] 0620 19:59:02.486319 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:02.610469 (+124150us) service_pool.cc:224] Handling call
0620 19:59:02.633884 (+ 23415us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:03.096476 (+462592us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:04.143651 (+1047175us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54000b95d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:04.164378 (+ 20727us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":928173952}
I20260620 19:59:04.439751 8479 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.429992 8407 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.473459 8636 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:04.450907 8604 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.478412 8581 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.518752 8648 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:04.521131 8634 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:04.477983 8566 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.525239 8496 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:04.526979 8548 consensus_meta.cc:287] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.989s user 0.007s sys 0.003s
I20260620 19:59:04.527474 8547 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.528117 8547 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.501938 8643 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.521263 8595 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:04.446537 8639 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.537734 8643 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.521263 8648 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:04.551002 8653 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.522025 8634 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.551788 8653 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.523874 8568 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:04.569444 8643 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:04.553505 8648 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:04.573079 8648 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:04.572988 8599 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:04.415411 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:04.750725 6231 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.000s sys 0.010s
W20260620 19:59:04.750973 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:04.751233 6231 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.001s sys 0.010s
W20260620 19:59:04.632114 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.651212 6303 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:04.652319 6307 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
W20260620 19:59:04.657593 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:04.658723 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.658861 6378 raft_consensus.cc:1240] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:04.678047 6374 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:04.678560 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:04.474148 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:04.679216 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 453854us
I20260620 19:59:04.680785 6911 raft_consensus.cc:1240] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:04.686780 8001 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:04.515789 8636 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.762065 6230 raft_consensus.cc:1240] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:04.763320 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:04.764148 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 591) took 1609 ms. Trace:
I20260620 19:59:04.764300 6378 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:04.764844 8636 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:04.765410 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:04.582858 8639 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:04.767401 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:04.768599 6905 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:04.762459 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:04.768581 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:04.771533 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:04.771641 8479 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:04.772746 6908 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 5
I20260620 19:59:04.773737 6375 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
W20260620 19:59:04.774370 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 60933.8us
I20260620 19:59:04.579478 8653 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:04.566733 8588 consensus_meta.cc:287] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.800s user 0.005s sys 0.007s
I20260620 19:59:04.745646 8432 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:04.742910 8595 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.746732 8502 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:04.746012 8643 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:04.793810 8574 consensus_meta.cc:287] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.071s user 0.007s sys 0.005s
I20260620 19:59:04.794405 8643 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.747853 8638 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.748672 8644 consensus_queue.cc:1059] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:59:04.751120 8496 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:04.639030 6302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1755) took 1467 ms. Trace:
I20260620 19:59:04.761039 8548 consensus_queue.cc:1059] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260620 19:59:04.772835 6374 raft_consensus.cc:1240] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:04.776103 6905 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:04.747368 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:04.784612 6381 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:04.778667 8432 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:04.786001 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:04.787537 8580 consensus_meta.cc:287] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.956s user 0.006s sys 0.005s
I20260620 19:59:04.789551 6877 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:04.790246 8595 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.791610 8502 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.793540 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:04.786201 8634 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:04.799424 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.926926 8604 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:04.941215 6373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1533) took 1245 ms (client timeout 1641 ms). Trace:
I20260620 19:59:04.942291 8345 raft_consensus.cc:3055] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
W20260620 19:59:04.942739 8650 log.cc:927] Time spent T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260620 19:59:04.942811 8345 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:04.944535 8345 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.946164 8345 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:04.947916 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 121721us
I20260620 19:59:04.948463 8502 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:04.949492 6875 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:04.901036 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:04.951803 8580 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:04.953815 6229 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1868) took 2052 ms (client timeout 1734 ms). Trace:
W20260620 19:59:04.954262 6229 rpcz_store.cc:269] 0620 19:59:02.901576 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.462830 (+561254us) service_pool.cc:224] Handling call
0620 19:59:04.953354 (+1490524us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b540004b5d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:04.953550 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":256120704}
I20260620 19:59:04.955735 6228 raft_consensus.cc:1240] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:04.764624 6305 rpcz_store.cc:276] 0620 19:59:03.154673 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.247769 (+ 93096us) service_pool.cc:224] Handling call
0620 19:59:04.763613 (+1515844us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54003255d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:04.763884 (+ 271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":295668096}
I20260620 19:59:04.957796 6228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2003) took 1017 ms. Trace:
I20260620 19:59:04.957968 8496 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:04.927026 8452 consensus_meta.cc:287] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 1.015s user 0.014s sys 0.002s
I20260620 19:59:04.958218 6228 rpcz_store.cc:276] 0620 19:59:03.940758 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.129810 (+189052us) service_pool.cc:224] Handling call
0620 19:59:04.955326 (+825516us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:04.956172 (+ 846us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:04.957554 (+ 1382us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b540004b5d0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:04.957733 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1893314816}
I20260620 19:59:04.975242 6908 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:04.961174 8452 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.803359 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:04.941793 6373 rpcz_store.cc:269] 0620 19:59:03.696274 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.698443 (+ 2169us) service_pool.cc:224] Handling call
0620 19:59:04.785653 (+1087210us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b5800142e98. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:59:04.941109 (+155456us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9228160}
I20260620 19:59:04.989080 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:04.989871 6908 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:04.839215 8659 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:04.998508 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 168935us
I20260620 19:59:04.998560 6902 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:05.008476 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.009307 8598 raft_consensus.cc:3055] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:05.009718 8598 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:05.010856 8598 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.012234 8598 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:05.011874 8537 consensus_queue.cc:260] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:05.013289 8537 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:05.014747 8566 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.015542 8566 raft_consensus.cc:3055] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:05.015992 8566 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:05.017187 8542 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.017112 8566 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:05.017830 8542 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:04.938385 6302 rpcz_store.cc:276] 0620 19:59:03.171456 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.101890 (+930434us) service_pool.cc:224] Handling call
0620 19:59:04.107205 (+ 5315us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:04.108467 (+ 1262us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:04.108494 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:04.108965 (+ 471us) log.cc:844] Serialized 26 byte log entry
0620 19:59:04.362877 (+253912us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:04.362937 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:04.363029 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:04.638397 (+275368us) raft_consensus.cc:1685] finished
0620 19:59:04.638462 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:04.638808 (+ 346us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":887,"raft.queue_time_us":972,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":57393,"threads_started":1,"wal-append.queue_time_us":399272}
I20260620 19:59:05.027208 6299 raft_consensus.cc:1240] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:05.043718 6377 raft_consensus.cc:1240] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260620 19:59:05.025713 6298 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1486) took 1874 ms (client timeout 1699 ms). Trace:
I20260620 19:59:05.026280 8443 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap starting.
I20260620 19:59:05.047075 8146 raft_consensus.cc:3055] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:05.047518 8648 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.049005 8432 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:05.023252 8574 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:05.046263 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.051024 8491 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.055740 8491 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:05.054455 8146 raft_consensus.cc:740] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:05.058135 8146 consensus_queue.cc:260] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.059449 8146 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:05.063066 8491 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:05.063855 8491 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:05.066020 8496 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.006511 8643 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:05.052590 8474 raft_consensus.cc:3055] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Stepping down as leader of term 5
W20260620 19:59:05.051301 6298 rpcz_store.cc:269] 0620 19:59:03.151251 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.156211 (+ 4960us) service_pool.cc:224] Handling call
0620 19:59:05.025206 (+1868995us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54001cd754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:05.025493 (+ 287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":526918528}
I20260620 19:59:05.069751 6345 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:05.032186 8452 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:04.998750 8638 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.002641 8580 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:05.067319 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 117991us
I20260620 19:59:05.735949 8465 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.169802 8533 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:05.736238 8576 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.736900 8465 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.737176 8576 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.121840 8583 consensus_queue.cc:1059] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.618s
W20260620 19:59:05.122227 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:05.407774 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:05.741735 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:05.138414 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.360630 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1882) took 1665 ms. Trace:
W20260620 19:59:05.165598 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:05.070237 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 280824us
I20260620 19:59:05.736286 6875 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:05.454420 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 588) took 2510 ms. Trace:
I20260620 19:59:05.132737 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:05.070765 8434 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
W20260620 19:59:05.525339 8598 consensus_meta.cc:287] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.513s user 0.007s sys 0.003s
I20260620 19:59:05.129877 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:05.105286 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:59:05.213119 8665 log.cc:927] Time spent T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260620 19:59:05.171406 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1753) took 2000 ms. Trace:
I20260620 19:59:05.191514 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 1999) took 1365 ms. Trace:
I20260620 19:59:05.754347 6300 rpcz_store.cc:276] 0620 19:59:02.944258 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:02.944440 (+ 182us) service_pool.cc:224] Handling call
0620 19:59:02.944808 (+ 368us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:03.622615 (+677807us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:05.453766 (+1831151us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b5400324bd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:05.454207 (+ 441us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1556937600}
W20260620 19:59:05.123031 6307 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 599) took 1355 ms (client timeout 1577 ms). Trace:
I20260620 19:59:05.069515 8618 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.762862 8465 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.794477 8592 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.762964 8434 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:05.804463 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 802989us
I20260620 19:59:05.765247 8547 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.306375 8436 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.765592 8597 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.066576 8595 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.067204 8474 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:05.771145 8541 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:05.131296 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 811) took 1951 ms. Trace:
I20260620 19:59:05.772598 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 451) took 1948 ms. Trace:
W20260620 19:59:05.773254 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 703393us
I20260620 19:59:05.773772 6198 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:05.773857 8457 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260620 19:59:05.776314 8537 consensus_meta.cc:287] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.763s user 0.014s sys 0.000s
I20260620 19:59:05.777309 8622 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:05.764595 6299 rpcz_store.cc:276] 0620 19:59:03.170965 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.099829 (+928864us) service_pool.cc:224] Handling call
0620 19:59:05.026783 (+926954us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:05.051081 (+ 24298us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:05.171129 (+120048us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b54001cd750. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:05.171316 (+ 187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2087234304}
I20260620 19:59:05.783367 8576 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.121660 8668 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.070578 8566 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:05.067041 8643 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:59:05.070502 8653 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:05.192967 8644 raft_consensus.cc:3055] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:05.466519 8466 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:05.135174 8599 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:05.764891 6230 rpcz_store.cc:276] 0620 19:59:03.825981 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.826134 (+ 153us) service_pool.cc:224] Handling call
0620 19:59:03.826430 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:04.762683 (+936253us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:05.191178 (+428495us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b5400367ed4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:05.191413 (+ 235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2149852416}
I20260620 19:59:05.795334 6911 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:05.360539 8581 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.529902 6902 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
W20260620 19:59:05.789529 6307 rpcz_store.cc:269] 0620 19:59:03.767656 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.859243 (+ 91587us) service_pool.cc:224] Handling call
0620 19:59:05.122821 (+1263578us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b54003a6454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:05.122965 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1778634624}
I20260620 19:59:05.749112 6225 rpcz_store.cc:276] 0620 19:59:03.695065 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.695738 (+ 673us) service_pool.cc:224] Handling call
0620 19:59:03.696005 (+ 267us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:03.719287 (+ 23282us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:03.719320 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:03.719849 (+ 529us) log.cc:844] Serialized 26 byte log entry
0620 19:59:04.949779 (+1229930us) spinlock_profiling.cc:243] Waited 7.61 ms on lock 0x7b1800305ce0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b8459471 00007fa6b8875def
0620 19:59:05.006855 (+ 57076us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:05.007670 (+ 815us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:05.360041 (+352371us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b54001e41d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:05.360123 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:05.360173 (+ 50us) raft_consensus.cc:1685] finished
0620 19:59:05.360189 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:05.360456 (+ 267us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18552,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":11661,"spinlock_wait_cycles":70307072,"thread_start_us":33600,"threads_started":1,"wal-append.queue_time_us":1286014}
I20260620 19:59:05.817574 6380 rpcz_store.cc:276] 0620 19:59:03.824366 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:05.129381 (+1305015us) service_pool.cc:224] Handling call
0620 19:59:05.772535 (+643154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:05.115695 8659 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:05.825347 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 89257us
W20260620 19:59:05.826380 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 52860.3us
I20260620 19:59:05.840898 6198 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:05.841756 8618 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:05.842170 8618 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:05.842572 8618 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:05.831102 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 67136.3us
I20260620 19:59:05.113037 8574 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:05.828725 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 57004us
I20260620 19:59:05.833874 8446 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260620 19:59:05.837395 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.822252 8648 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:05.842283 6198 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:05.866772 8576 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:05.888343 6875 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:05.828608 8593 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:05.867993 8580 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.868139 8606 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.868613 6345 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:05.868728 8654 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:05.837088 8547 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.869179 8597 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.836884 8592 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.828894 8653 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:05.906026 8446 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:05.870035 8635 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:05.870870 8542 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:05.892060 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.891642 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 621) took 1114 ms. Trace:
W20260620 19:59:05.867419 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:05.911907 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:05.912235 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:05.897099 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.830750 8644 raft_consensus.cc:740] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:05.913960 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.901373 8457 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.836966 8617 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:05.870460 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 616) took 1095 ms. Trace:
I20260620 19:59:05.910795 6911 rpcz_store.cc:276] 0620 19:59:04.777536 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.793069 (+ 15533us) service_pool.cc:224] Handling call
0620 19:59:05.891473 (+1098404us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54004f8dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:05.891585 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1968049920}
I20260620 19:59:05.875454 8659 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.898546 8593 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.874159 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 617) took 1098 ms. Trace:
I20260620 19:59:05.869211 8622 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:05.907140 8653 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:05.878343 8638 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:05.914388 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.870956 8582 consensus_queue.cc:1059] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
W20260620 19:59:05.915887 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 73687.7us
W20260620 19:59:05.870518 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 46764.3us
I20260620 19:59:05.835498 8434 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:05.870630 8436 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.930783 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:05.871603 8668 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:05.935173 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:05.817312 6903 rpcz_store.cc:276] 0620 19:59:03.179298 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:03.202369 (+ 23071us) service_pool.cc:224] Handling call
0620 19:59:03.262815 (+ 60446us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:03.664175 (+401360us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:03.665230 (+ 1055us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:03.668595 (+ 3365us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:03.670218 (+ 1623us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:05.130745 (+1460527us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400491e54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:05.130824 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:05.131084 (+ 260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920436096}
I20260620 19:59:05.938674 6273 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:05.875139 8474 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.939234 6877 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:05.939916 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:05.922253 6902 rpcz_store.cc:276] 0620 19:59:04.775420 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.802828 (+ 27408us) service_pool.cc:224] Handling call
0620 19:59:05.870362 (+1067534us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:05.921836 8635 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.944097 6377 raft_consensus.cc:1217] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 LEADER]: Deduplicated request from leader. Original: 6.8->[7.9-8.10] Dedup: 7.9->[8.10-8.10]
I20260620 19:59:05.929543 8617 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:05.945232 6377 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: [7.9-8.10]
I20260620 19:59:05.916407 8547 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:05.937666 8537 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:05.939930 8452 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:05.950881 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 120737us
I20260620 19:59:05.923782 8654 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.941504 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:05.955722 6374 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:05.958849 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 70705.2us
I20260620 19:59:05.933189 8606 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.953655 8638 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:05.957221 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:05.953907 8659 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:05.934886 8457 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.967950 8675 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:05.968745 8573 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:05.968513 8675 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.953179 8465 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:05.957978 8474 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:05.974329 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:05.985440 8593 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:05.965378 8638 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:05.986235 8638 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:05.953759 8452 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:05.962976 8537 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:05.966989 8659 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:05.977698 8675 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.011564 6875 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:06.011855 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.007382 6273 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:05.966662 8644 consensus_queue.cc:260] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.000168 8659 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:06.019585 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.016902 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:05.983587 8465 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.000715 6373 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:06.024981 6381 raft_consensus.cc:1275] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260620 19:59:06.027040 6877 leader_election.cc:400] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5.
I20260620 19:59:06.027786 6877 leader_election.cc:403] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:59:05.927039 6906 rpcz_store.cc:276] 0620 19:59:04.775842 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.786580 (+ 10738us) service_pool.cc:224] Handling call
0620 19:59:04.787245 (+ 665us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:05.058572 (+271327us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:05.058632 (+ 60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:05.059329 (+ 697us) log.cc:844] Serialized 26 byte log entry
0620 19:59:05.175800 (+116471us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:05.175875 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:05.175981 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:05.873775 (+697794us) raft_consensus.cc:1685] finished
0620 19:59:05.873802 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:05.873977 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":270483,"raft.queue_time_us":270645,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":115895,"threads_started":1,"wal-append.queue_time_us":726210}
I20260620 19:59:06.031775 6374 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:06.031886 8644 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:06.032292 8345 consensus_queue.cc:1059] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:06.027510 6381 pending_rounds.cc:85] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 1
I20260620 19:59:05.995970 8598 consensus_queue.cc:1059] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260620 19:59:06.042759 8446 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:06.004118 8533 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
I20260620 19:59:06.045092 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:06.045938 6902 raft_consensus.cc:2393] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:06.046669 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:06.048828 8452 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.016453 8574 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:06.051199 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 77894.4us
I20260620 19:59:06.035882 6381 pending_rounds.cc:107] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260620 19:59:06.052227 6381 raft_consensus.cc:2889] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:06.057840 8679 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:06.058176 8457 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.058420 8679 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.060746 8679 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.061201 8679 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.061666 8679 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.062263 6378 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:06.028376 8668 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:06.067062 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 60493.4us
I20260620 19:59:06.052507 8566 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:06.029953 8592 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.067763 6273 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:06.055409 8675 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:06.073167 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 151770us
I20260620 19:59:06.065099 8597 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:06.081359 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 75053.2us
I20260620 19:59:06.068140 8598 consensus_queue.cc:1243] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260620 19:59:06.066511 8446 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5.
W20260620 19:59:06.073589 8677 log.cc:927] Time spent T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20260620 19:59:06.064370 8576 raft_consensus.cc:697] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:06.085944 6345 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:06.089980 8592 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260620 19:59:06.090683 8676 log.cc:927] Time spent T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260620 19:59:06.098766 8434 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.098677 8683 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.076606 8654 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.071583 8457 consensus_queue.cc:1059] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260620 19:59:06.102828 8635 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.104127 8465 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.108032 8606 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.075798 8622 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.091655 8436 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.127043 6877 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:59:06.134109 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 66401.2us
I20260620 19:59:06.112839 8547 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
W20260620 19:59:06.142179 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 71926.9us
I20260620 19:59:06.146090 6875 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:06.083776 8648 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260620 19:59:06.090699 8574 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.170976 6379 raft_consensus.cc:2410] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260620 19:59:06.126981 8683 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.142966 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:06.174436 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 61299.5us
I20260620 19:59:06.128110 8542 raft_consensus.cc:3055] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:06.173998 6344 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:06.142869 8541 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.070433 8617 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.082903 8537 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.161203 8598 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:06.160526 8599 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.160774 8434 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:06.183060 8691 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.177750 8583 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:06.189505 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:06.189653 6198 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.186534 8541 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.188720 8583 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:06.192271 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 134444us
I20260620 19:59:06.193101 8541 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.189126 8622 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.195099 8622 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.195716 8618 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.197175 8622 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.197734 8622 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.198848 8599 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.200048 6230 raft_consensus.cc:1240] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260620 19:59:06.201164 8599 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.201259 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:06.179401 8542 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:06.203158 6228 raft_consensus.cc:1240] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:06.203349 8434 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.203528 8542 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.204597 8683 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.204749 6232 raft_consensus.cc:1240] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:06.204885 8542 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:06.205700 8434 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.205873 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:06.206128 6877 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:06.206781 6229 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:06.207583 8688 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:06.208293 6227 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 10. (index mismatch)
I20260620 19:59:06.183230 8576 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.209810 6224 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260620 19:59:06.210747 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.210879 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 619) took 1434 ms. Trace:
I20260620 19:59:06.211791 6225 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:06.212805 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:06.190351 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 76670.3us
I20260620 19:59:06.221540 6378 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:06.240446 8688 raft_consensus.cc:697] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:06.198153 8622 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.198372 8618 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:06.259125 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 69826.5us
I20260620 19:59:06.212033 8679 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.249276 8345 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.251538 8683 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.282801 8690 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.282950 8345 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.283046 8683 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.283373 8690 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.285127 8345 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.285319 8690 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.285730 8659 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:06.286175 6378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 1896) took 1627 ms (client timeout 1934 ms). Trace:
W20260620 19:59:06.286582 6378 rpcz_store.cc:269] 0620 19:59:04.658360 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.768212 (+109852us) service_pool.cc:224] Handling call
0620 19:59:06.286010 (+1517798us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400121bd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:06.286113 (+ 103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2968156416}
I20260620 19:59:06.287802 8659 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.203191 8653 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.289274 8638 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.289785 8653 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.211270 6909 rpcz_store.cc:276] 0620 19:59:04.776724 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.987522 (+210798us) service_pool.cc:224] Handling call
0620 19:59:04.999473 (+ 11951us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:05.752044 (+752571us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:05.752087 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:05.768136 (+ 16049us) log.cc:844] Serialized 26 byte log entry
0620 19:59:05.839599 (+ 71463us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:05.839644 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:05.839739 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:06.210664 (+370925us) raft_consensus.cc:1685] finished
0620 19:59:06.210685 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:06.210821 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":620293,"raft.queue_time_us":643081,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":3225,"thread_start_us":70963,"threads_started":1,"wal-append.queue_time_us":157947}
I20260620 19:59:06.206348 8573 consensus_queue.cc:1059] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260620 19:59:06.291594 8622 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.293041 8533 raft_consensus.cc:3055] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:06.293538 8533 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:06.187047 8574 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.294512 8533 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.295879 8533 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:06.296217 8574 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.296904 8592 consensus_queue.cc:1059] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:59:06.297279 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:06.297596 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.297851 8592 raft_consensus.cc:3055] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:06.298027 6223 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:06.298272 8592 raft_consensus.cc:740] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:06.299325 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.299222 8592 consensus_queue.cc:260] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.299891 6226 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:06.300418 8592 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:06.302405 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:06.266746 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.189092 8635 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:06.238314 8654 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:06.276228 8598 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:06.281323 8675 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:06.281725 8630 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.212623 8691 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.303104 6226 raft_consensus.cc:2393] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:06.276171 6379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 1889) took 1826 ms (client timeout 2001 ms). Trace:
I20260620 19:59:06.241837 8688 consensus_queue.cc:237] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:06.311082 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.276657 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.305017 6875 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:06.345924 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.285233 8683 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:06.341181 6379 rpcz_store.cc:269] 0620 19:59:04.449712 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:05.132216 (+682504us) service_pool.cc:224] Handling call
0620 19:59:06.275987 (+1143771us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400062a54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:06.276115 (+ 128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920730496}
I20260620 19:59:06.321167 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:06.349046 6227 raft_consensus.cc:2410] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:06.321808 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.322921 6230 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
I20260620 19:59:06.323424 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.350348 6232 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:06.323995 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:06.324676 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 118651us
W20260620 19:59:06.326483 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.328837 8573 consensus_queue.cc:1243] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260620 19:59:06.333521 8638 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:06.334772 8659 consensus_queue.cc:1059] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:06.340688 6225 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
W20260620 19:59:06.345943 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.345046 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:06.358831 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:06.392443 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:06.392869 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.393803 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:06.394675 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.395442 6223 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:06.317024 8436 consensus_queue.cc:1059] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260620 19:59:06.341296 8345 consensus_queue.cc:1059] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20260620 19:59:06.341552 8630 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:06.660403 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.661900 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:06.718860 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.395678 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:06.621760 6271 connection.cc:256] RPC call timeout handler was delayed by -1.2092158099999324s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
W20260620 19:59:06.937140 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.937355 6380 raft_consensus.cc:2410] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:06.415057 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:06.512126 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 50025.3us
I20260620 19:59:06.442119 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:06.420089 6228 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
W20260620 19:59:06.875907 8592 consensus_meta.cc:287] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.575s user 0.009s sys 0.002s
I20260620 19:59:06.396112 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:06.938135 6224 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:06.940260 6374 raft_consensus.cc:2393] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:06.638861 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.685377 8574 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:06.429793 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:06.949004 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:06.949221 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.939455 6375 raft_consensus.cc:2393] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:06.415351 8598 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.941769 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:06.729663 6196 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 2.831s (SENT) [suppressed 30 similar messages]
I20260620 19:59:06.365751 8457 consensus_queue.cc:1059] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.577s
I20260620 19:59:06.935513 6226 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
W20260620 19:59:06.945977 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:06.613600 8693 log.cc:927] Time spent T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
W20260620 19:59:06.937824 8617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.008s sys 0.015s
W20260620 19:59:06.951270 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.359673 8644 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.950340 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:06.508299 8573 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.519654 8700 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.963235 8700 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.963879 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.964775 6229 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 2.
I20260620 19:59:06.965473 8700 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.966859 8573 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.968987 8573 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:06.969548 8644 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:06.971750 8644 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.973776 8644 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.974941 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:06.976598 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.954777 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:06.957167 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 227478us
I20260620 19:59:06.638911 8547 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.696612 8682 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:06.959162 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.986088 6226 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:06.959936 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:06.960727 8617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.009s sys 0.015s
I20260620 19:59:06.986935 6227 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:06.621762 8583 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:06.950150 6372 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:06.961573 6377 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:06.962225 8541 consensus_queue.cc:1059] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260620 19:59:06.979014 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.983037 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:06.974261 8644 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.981170 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.984694 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:06.985208 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.016527 6229 raft_consensus.cc:2393] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 8: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:07.008716 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:06.988855 8573 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:07.008731 8547 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.008745 8704 consensus_queue.cc:1059] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:07.011003 6307 raft_consensus.cc:1240] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:07.012463 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:07.013530 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:07.016422 6230 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:07.022411 6300 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:06.984688 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.014875 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.023221 6302 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:07.021653 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:06.952598 8574 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:07.020159 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.020812 6229 raft_consensus.cc:1240] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:07.024660 6224 raft_consensus.cc:2410] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:07.030942 6227 raft_consensus.cc:1275] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:59:07.035154 8707 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:07.035732 8707 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.037822 8707 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.038681 8606 raft_consensus.cc:3055] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:07.039088 8606 raft_consensus.cc:740] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:07.040397 8704 raft_consensus.cc:3055] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:07.040776 8704 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:07.041574 8704 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.042943 8547 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.043689 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 620) took 2266 ms. Trace:
I20260620 19:59:07.044561 8644 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:07.023365 8573 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.040014 8606 consensus_queue.cc:260] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:06.988795 8436 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:07.066764 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 51482.2us
I20260620 19:59:07.070518 6345 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.072718 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.073493 6305 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:07.024397 6223 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:07.021436 8457 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260620 19:59:07.066970 6298 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:07.086723 8704 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:07.088195 8574 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.089395 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.089972 8542 raft_consensus.cc:3055] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:07.090274 6373 raft_consensus.cc:2393] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:07.090391 8542 raft_consensus.cc:740] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:07.090832 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 816389us
W20260620 19:59:07.091212 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 465719us
W20260620 19:59:07.091922 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.091555 8542 consensus_queue.cc:260] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.092080 8654 consensus_queue.cc:1059] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:07.092701 8542 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:07.092993 8654 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:07.093391 8654 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:07.093788 8654 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:07.094753 8474 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:06.989202 8682 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.095940 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:07.096989 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 681862us
I20260620 19:59:07.097188 8682 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.097734 6877 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:07.098068 8573 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.098471 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.098976 6877 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:07.044059 6905 rpcz_store.cc:276] 0620 19:59:04.777136 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:04.996301 (+219165us) service_pool.cc:224] Handling call
0620 19:59:05.042786 (+ 46485us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:05.071037 (+ 28251us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:05.071075 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:05.211008 (+139933us) log.cc:844] Serialized 26 byte log entry
0620 19:59:05.859629 (+648621us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:06.109560 (+249931us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b18004317a0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c5c55 00007fa6bc2b4247 00007fa6b88c4d87 00007fa6b88d449f 00007fa6b88d4418
0620 19:59:06.186411 (+ 76851us) log.cc:844] Serialized 14 byte log entry
0620 19:59:07.043169 (+856758us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b18004317a0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b574340a 00007fa6bee885d9 00007fa6b57cb240 00007fa6b57c5ce4 00007fa6b57c5c55 00007fa6b88b2440 00007fa6b88c4af6 00007fa6b88d5667 00007fa6b88d55e8
0620 19:59:07.043344 (+ 175us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:07.043427 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:07.043465 (+ 38us) raft_consensus.cc:1685] finished
0620 19:59:07.043482 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:07.043611 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22798,"prepare.queue_time_us":356173,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":27129,"raft.queue_time_us":836518,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":2979,"spinlock_wait_cycles":32098560,"thread_start_us":145635,"threads_started":2,"wal-append.queue_time_us":711896}
W20260620 19:59:07.099870 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 475347us
I20260620 19:59:07.064071 8680 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:07.103060 8680 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:07.103767 8680 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:07.032491 8708 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:07.104595 8708 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.068795 8653 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:07.082412 8452 raft_consensus.cc:3055] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:07.075448 8635 raft_consensus.cc:697] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:07.106916 8452 raft_consensus.cc:740] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:07.107801 8635 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.106917 8653 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.111064 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.111805 8700 raft_consensus.cc:3055] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:07.075775 8688 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260620 19:59:07.112205 8700 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:07.082962 8457 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:07.113659 8688 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:07.094944 8638 raft_consensus.cc:3055] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:07.081712 6231 raft_consensus.cc:1240] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:07.107844 8452 consensus_queue.cc:260] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:07.084340 8643 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:07.083928 6305 raft_consensus.cc:1275] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260620 19:59:07.110512 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.119562 6875 connection.cc:256] RPC call timeout handler was delayed by -2.3297698800000717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
I20260620 19:59:07.129374 8537 raft_consensus.cc:3055] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:07.129561 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.129822 8537 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:07.130481 6307 raft_consensus.cc:2410] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:07.130708 8537 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:07.131901 6345 leader_election.cc:336] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 1.706s (SENT)
I20260620 19:59:07.131667 8683 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:07.132351 6345 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:07.132691 6875 connection.cc:256] RPC call timeout handler was delayed by -2.2550698800000646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21850000000000003s for a response
I20260620 19:59:07.133224 6229 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260620 19:59:07.133201 6875 connection.cc:256] RPC call timeout handler was delayed by -1.6421698800000968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2866s for a response
W20260620 19:59:07.135360 6875 connection.cc:256] RPC call timeout handler was delayed by -1.4693698800000448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3058s for a response
I20260620 19:59:07.123051 8457 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.141814 8457 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.123356 8708 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.142686 8688 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.144940 8688 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.143081 8712 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:07.146677 6271 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:07.159276 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.113055 8700 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:07.161881 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.173939 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.162922 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.163611 6909 raft_consensus.cc:1240] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:07.163533 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.164049 8708 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:07.164706 6902 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:07.165323 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.165920 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.207544 6378 raft_consensus.cc:2410] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:07.199676 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.166781 6379 raft_consensus.cc:1240] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:07.166857 6905 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:46646
I20260620 19:59:07.167906 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.168555 8691 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.169236 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.126094 8653 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:07.125502 8638 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:07.133263 8682 consensus_queue.cc:1048] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260620 19:59:07.174564 6380 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:07.175686 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.176002 6907 raft_consensus.cc:1240] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:07.235510 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.236219 6907 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:07.237303 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:07.237885 6907 raft_consensus.cc:2376] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6. Re-sending same reply.
I20260620 19:59:07.153813 6372 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:07.174626 6375 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:07.147076 8711 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:07.234259 6305 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:07.208562 6908 raft_consensus.cc:2410] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:07.240829 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.153182 8712 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.126210 8436 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.241617 6307 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:59:07.223291 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 83735us
I20260620 19:59:07.162668 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.170465 8643 raft_consensus.cc:697] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:07.240936 6911 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:59:07.230520 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 104725us
I20260620 19:59:07.242897 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:07.244588 6909 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:07.209482 6904 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:07.246794 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:07.247643 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.248539 6198 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.233719 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.255262 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.257081 6908 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:07.260391 6907 raft_consensus.cc:2393] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:07.261456 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.258589 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.261368 6198 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.264053 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.243604 8712 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:07.267180 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.242089 6903 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:07.269989 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.250756 8700 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:07.267771 6305 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
I20260620 19:59:07.274098 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:07.275148 6227 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:59:07.275359 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.276050 6907 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:07.276985 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.278079 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.280445 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.281035 6225 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:07.265072 8691 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:07.283964 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.286263 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.286756 6305 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:07.284567 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.287552 6911 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:07.294334 6907 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:07.282819 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.281840 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.281527 8708 consensus_queue.cc:1059] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260620 19:59:07.288503 8691 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.312090 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:07.321283 6225 raft_consensus.cc:2376] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5. Re-sending same reply.
I20260620 19:59:07.295513 8682 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:07.294232 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.285035 8638 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.304600 8643 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.304868 6223 raft_consensus.cc:1240] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:07.345321 6910 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260620 19:59:07.520190 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 258823us
W20260620 19:59:07.505378 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 221249us
W20260620 19:59:07.485581 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.363628 6228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 356) took 1067 ms. Trace:
I20260620 19:59:07.522745 6228 rpcz_store.cc:276] 0620 19:59:06.295760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:06.304038 (+ 8278us) service_pool.cc:224] Handling call
0620 19:59:07.363461 (+1059423us) spinlock_profiling.cc:243] Waited 70.4 ms on lock 0x7b5400245c54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:07.363572 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161897216}
I20260620 19:59:07.364004 6345 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:07.398121 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.524753 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.525202 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.525668 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.526324 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.365208 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.527226 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.534842 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.489481 8443 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:59:07.333674 8691 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:07.343973 6904 raft_consensus.cc:2410] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260620 19:59:07.540702 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.547695 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.541006 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.541951 6909 raft_consensus.cc:2410] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:07.350104 6372 raft_consensus.cc:1240] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260620 19:59:07.554549 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.547148 8474 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:07.364331 6902 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:07.557974 8474 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.555287 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.336910 8682 raft_consensus.cc:697] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:07.400910 8718 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260620 19:59:07.545248 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 264980us
I20260620 19:59:07.548861 6911 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:07.550477 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:07.551219 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.362149 6298 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:07.554065 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.566900 6305 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:07.554677 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.567746 6228 raft_consensus.cc:2393] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:07.555600 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.546465 6906 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:07.556304 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.556382 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:07.495409 8715 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.154s user 0.000s sys 0.000s
I20260620 19:59:07.556783 8583 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:07.544214 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.558486 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.557241 8711 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:07.575050 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.501044 8436 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.580425 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.581180 6225 raft_consensus.cc:2393] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:07.456650 8654 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.536429 6905 raft_consensus.cc:606] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:07.401302 8713 log.cc:927] Time spent T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260620 19:59:07.371483 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.554320 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.586202 6905 raft_consensus.cc:614] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:07.570434 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.557060 6271 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:07.572171 6299 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:07.560261 8474 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:07.573717 8583 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.578109 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:07.578105 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:07.590217 6228 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:07.555287 6904 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 2.
W20260620 19:59:07.581691 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 217765us
I20260620 19:59:07.576426 6196 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.584216 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.591269 6305 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:07.591421 6196 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.591681 8721 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:07.592231 8721 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.600477 8721 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:07.593170 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.608839 6301 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:59:07.610565 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.612119 8703 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.665s user 0.004s sys 0.005s
I20260620 19:59:07.594264 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.615263 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.615944 6224 raft_consensus.cc:2393] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:07.616681 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.620453 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1885) took 1324 ms. Trace:
I20260620 19:59:07.620831 6231 rpcz_store.cc:276] 0620 19:59:06.296377 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:06.318058 (+ 21681us) service_pool.cc:224] Handling call
0620 19:59:07.081551 (+763493us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:07.128256 (+ 46705us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:07.620273 (+492017us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b5400369f50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:07.620397 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1714848128}
I20260620 19:59:07.621985 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:07.622303 8436 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
W20260620 19:59:07.622442 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 65439.1us
I20260620 19:59:07.622922 8436 raft_consensus.cc:697] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:07.623854 8443 tablet_bootstrap.cc:492] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Bootstrap complete.
I20260620 19:59:07.624636 8443 ts_tablet_manager.cc:1403] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent bootstrapping tablet: real 2.599s user 0.029s sys 0.032s
I20260620 19:59:07.586572 8654 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:07.625550 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:07.632694 6908 raft_consensus.cc:2393] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:07.625150 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:07.596375 8583 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:07.599380 8682 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.634359 8592 raft_consensus.cc:3055] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:07.634749 8592 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:07.635619 8592 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:07.636988 8592 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:07.631084 8708 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260620 19:59:07.639667 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.639776 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.640264 6907 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:07.640472 6908 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:07.641184 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.641575 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.642454 6908 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:07.643467 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.644162 6908 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:07.645043 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.645723 6908 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:07.623860 8436 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.650763 6906 raft_consensus.cc:2376] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5. Re-sending same reply.
I20260620 19:59:07.651970 8537 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:07.652092 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.653368 6906 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:07.654491 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.655191 6906 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:07.656199 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.657485 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.658149 6906 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:07.660035 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.661254 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.661530 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:07.583655 6902 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:07.625227 8722 consensus_queue.cc:1059] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260620 19:59:07.663847 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:07.582451 8638 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:07.673743 6225 raft_consensus.cc:2410] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:07.666971 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.667438 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.668104 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:07.668769 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.678192 6378 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:07.669428 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260620 19:59:07.678946 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.679579 6377 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:07.679852 8692 raft_consensus.cc:3055] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:07.680375 8692 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:07.681494 8692 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:07.677552 6223 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260620 19:59:07.685483 6911 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260620 19:59:07.686012 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.662302 6904 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:07.672196 8606 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:07.672241 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.669382 6345 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:07.680409 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:07.686477 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.641850 6907 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:07.687322 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.687666 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.688555 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:07.690871 6345 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:07.691219 8541 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:07.691622 8598 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:07.691722 8630 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:07.692119 8598 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:07.693720 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:07.696641 8598 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:07.694957 6345 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:07.692468 8630 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:07.698827 8630 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.699291 8630 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:07.699419 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.700475 6910 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:07.701946 6910 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:07.705370 8452 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:07.706920 8697 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:07.707345 8697 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.707659 8697 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:07.693977 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.717410 6345 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:07.718374 6345 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:07.719183 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.719184 6345 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:07.720382 8702 log.cc:927] Time spent T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.780s user 0.002s sys 0.015s
I20260620 19:59:07.719342 8598 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:07.722935 8598 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:07.722774 8345 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:07.723371 8598 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:07.686764 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:07.726930 6908 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:07.734145 6375 raft_consensus.cc:2393] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:07.722728 8533 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:07.738998 8711 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:07.741916 8533 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.739588 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:07.741211 6228 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:07.743696 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:07.745388 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.713971 8692 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:07.723714 8345 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.747845 6907 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260620 19:59:07.749441 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 446681us
I20260620 19:59:07.745918 6375 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:07.751120 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.757046 6198 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.742234 8711 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.764174 6381 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:07.751632 6375 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:07.772166 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.765094 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:07.774816 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.742383 8533 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:07.776613 6907 raft_consensus.cc:1240] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:07.767231 6875 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.742547 8541 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260620 19:59:07.780608 8654 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:07.785423 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:07.796212 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.796830 6305 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
W20260620 19:59:07.777595 8704 consensus_meta.cc:287] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.690s user 0.013s sys 0.000s
I20260620 19:59:07.777096 6903 raft_consensus.cc:2410] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:07.763600 8345 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:07.710191 8443 raft_consensus.cc:359] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:07.776118 8711 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:07.776894 8700 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:07.830724 8618 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:07.839136 8618 raft_consensus.cc:697] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:07.840091 8618 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.841744 6374 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:07.835843 6903 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:07.844696 6305 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:59:07.846585 6907 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:59:07.849128 6378 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:07.851393 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:07.857070 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.858970 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.859673 6299 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:07.860540 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:07.861085 6307 raft_consensus.cc:2376] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7. Re-sending same reply.
I20260620 19:59:07.861912 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.863018 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.863586 6300 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:07.864459 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.865026 6303 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:07.866539 6375 raft_consensus.cc:2393] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:07.865779 6376 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:07.869422 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:07.870508 6378 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:07.872128 6377 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:07.873143 8474 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:07.873660 8474 raft_consensus.cc:697] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:07.873584 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.874505 6304 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:07.858266 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:07.858134 8533 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.881474 8700 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:07.881925 8700 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:07.882071 8533 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:07.883031 8712 consensus_meta.cc:287] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.604s user 0.009s sys 0.001s
I20260620 19:59:07.883590 6378 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:07.883423 8712 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.875134 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:07.885957 6306 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:07.898790 6375 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260620 19:59:07.899209 6375 pending_rounds.cc:85] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 6
I20260620 19:59:07.899434 6375 pending_rounds.cc:107] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260620 19:59:07.899650 6375 raft_consensus.cc:2889] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:07.841619 8443 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Initialized, Role: LEARNER
I20260620 19:59:07.825865 8654 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:07.918390 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 326913us
W20260620 19:59:07.921700 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.850437 6373 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:07.922680 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:07.923566 6372 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:07.924444 8721 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:07.926519 8691 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:07.935098 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:07.962808 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 339491us
I20260620 19:59:07.920051 6196 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:07.967303 8474 consensus_queue.cc:237] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:07.989372 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.888779 8598 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:08.031702 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:07.926393 8345 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:08.043578 6344 leader_election.cc:336] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.577s (SENT)
I20260620 19:59:07.961052 8630 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:08.174530 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 434893us
W20260620 19:59:08.174842 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 254800us
I20260620 19:59:08.175455 6196 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:08.174204 6224 raft_consensus.cc:2368] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7.
W20260620 19:59:08.181485 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.182963 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.183481 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.183763 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.165472 6874 leader_election.cc:336] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.577s (SENT)
I20260620 19:59:08.177166 6196 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:08.189518 6874 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:08.202271 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 445176us
W20260620 19:59:08.202739 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.203684 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.204227 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.206674 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:08.207495 6910 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:08.128046 6905 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:08.014094 8443 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:08.204948 6345 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:08.129755 8643 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:08.199702 8682 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260620 19:59:08.237989 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 72562.7us
I20260620 19:59:08.239810 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:08.240101 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.225026 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 213279us
W20260620 19:59:08.226337 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.238198 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:08.249325 6232 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
W20260620 19:59:08.257689 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 52800.5us
I20260620 19:59:08.259479 8654 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:08.260139 8654 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:08.267027 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.267866 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:08.271099 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:08.272119 6223 raft_consensus.cc:2410] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:08.272327 6225 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:08.273105 8541 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:08.273605 8541 raft_consensus.cc:697] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:08.274828 6903 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260620 19:59:08.235072 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 57971.4us
I20260620 19:59:08.275923 8733 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:08.276233 6196 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:08.276609 8733 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:08.276010 8630 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.274471 8541 consensus_queue.cc:237] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:08.213269 8436 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:08.261170 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.277004 8733 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:08.278088 8436 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:08.281587 8711 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.282518 8436 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:08.282776 6378 raft_consensus.cc:1240] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:08.282855 8749 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:08.283564 8749 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.285140 8726 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:08.236088 8576 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260620 19:59:08.285809 8726 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:08.172569 8697 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.213830 8712 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:08.295270 6344 leader_election.cc:336] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.191s (SENT)
I20260620 19:59:08.280280 6271 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:08.227962 6198 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:08.290450 8721 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.292268 8736 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260620 19:59:08.298902 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 71030us
W20260620 19:59:08.305081 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.295786 6344 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:08.250270 8682 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:08.319866 6225 raft_consensus.cc:1240] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:08.328401 8726 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:08.329828 8726 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:08.330210 8726 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:08.330291 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:08.331224 6379 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:08.324689 6198 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:08.257061 8730 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:08.334559 6380 raft_consensus.cc:2368] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5.
W20260620 19:59:08.304869 8592 consensus_meta.cc:287] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.667s user 0.011s sys 0.002s
I20260620 19:59:08.285835 8749 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.336840 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 788) took 1097 ms. Trace:
I20260620 19:59:08.325168 8682 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:08.339120 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 43873.4us
I20260620 19:59:08.348076 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:08.358294 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.373124 6195 leader_election.cc:336] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.699s (SENT)
I20260620 19:59:08.357422 6225 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:08.339848 6911 rpcz_store.cc:276] 0620 19:59:07.239861 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.715315 (+475454us) service_pool.cc:224] Handling call
0620 19:59:07.715706 (+ 391us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:07.739619 (+ 23913us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:07.739661 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:07.840702 (+101041us) spinlock_profiling.cc:243] Waited 924 us on lock 0x7b2c000125b8. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b71b3c71 00007fa6b71c35d2 00007fa6b88f5de1 00007fa6b88ba81a 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22
0620 19:59:07.841096 (+ 394us) log.cc:844] Serialized 26 byte log entry
0620 19:59:08.251073 (+409977us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:08.251425 (+ 352us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:08.251677 (+ 252us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:08.304760 (+ 53083us) raft_consensus.cc:1685] finished
0620 19:59:08.304785 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:08.336781 (+ 31996us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14649,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":5431,"spinlock_wait_cycles":2124544,"thread_start_us":395439,"threads_started":1,"wal-append.queue_time_us":451640}
I20260620 19:59:08.333323 8740 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:08.388659 6195 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:08.336750 8737 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:08.294185 8733 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.287062 8643 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:08.360381 6902 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.006s sys 0.003s
I20260620 19:59:08.337335 8730 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:08.340449 8712 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:59:08.324391 8700 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.347947 8537 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260620 19:59:08.513238 6299 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:08.510412 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:08.380677 8683 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:08.510787 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.619596 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:08.532084 6907 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 10. (index mismatch)
I20260620 19:59:08.509493 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.616919 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:08.681622 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 797) took 1064 ms. Trace:
I20260620 19:59:08.668272 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:08.582544 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:59:08.682667 8683 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:08.683547 8683 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:08.514456 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.684672 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.685024 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.507643 8751 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:08.686288 8751 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.686772 8700 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.512225 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:08.413043 8576 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.275s
I20260620 19:59:08.688052 6302 raft_consensus.cc:2393] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:59:08.435957 8725 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.646s user 0.004s sys 0.000s
I20260620 19:59:08.689062 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.689563 6229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2030) took 1443 ms. Trace:
I20260620 19:59:08.689845 6302 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
W20260620 19:59:08.680966 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 307784us
W20260620 19:59:08.690999 6902 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.907s user 0.011s sys 0.010s
I20260620 19:59:08.689946 6229 rpcz_store.cc:276] 0620 19:59:07.246572 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.246799 (+ 227us) service_pool.cc:224] Handling call
0620 19:59:07.247085 (+ 286us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:07.257813 (+ 10728us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:07.257859 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:07.279609 (+ 21750us) log.cc:844] Serialized 26 byte log entry
0620 19:59:07.805613 (+526004us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:07.805691 (+ 78us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:07.805780 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:08.435409 (+629629us) spinlock_profiling.cc:243] Waited 8.44 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:08.689315 (+253906us) raft_consensus.cc:1685] finished
0620 19:59:08.689338 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:08.689512 (+ 174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":926,"raft.queue_time_us":4519,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":6182,"spinlock_wait_cycles":19412352,"thread_start_us":471317,"threads_started":1,"wal-append.queue_time_us":485371}
I20260620 19:59:08.618556 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 372) took 1066 ms. Trace:
I20260620 19:59:08.434463 8697 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.692560 6224 rpcz_store.cc:276] 0620 19:59:07.552138 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.616457 (+ 64319us) service_pool.cc:224] Handling call
0620 19:59:08.618107 (+1001650us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54001414d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:08.618378 (+ 271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":709003520}
I20260620 19:59:08.423378 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:08.436403 6223 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:08.511778 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.394718 8737 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.695452 6303 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:08.514477 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:08.348798 8726 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:08.681221 6228 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 8.
I20260620 19:59:08.701287 8737 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.357522 8721 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:08.702052 8721 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:08.702493 8721 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:08.582021 8542 raft_consensus.cc:3055] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:08.703655 8542 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:08.704756 8542 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:08.341143 8750 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:08.706029 8542 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:08.706418 8750 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:08.707108 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.707794 8750 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:08.435734 8733 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:08.473160 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.712518 6231 raft_consensus.cc:2393] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:08.573321 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 234988us
I20260620 19:59:08.714375 6345 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:08.394932 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.716583 6304 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:08.718585 6307 raft_consensus.cc:2393] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:08.682737 6908 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:08.357410 8728 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:08.682235 6907 rpcz_store.cc:276] 0620 19:59:07.617262 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.879427 (+262165us) service_pool.cc:224] Handling call
0620 19:59:08.531410 (+651983us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:08.531652 (+ 242us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:08.681352 (+149700us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:08.681568 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10315520}
W20260620 19:59:08.569020 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 234151us
I20260620 19:59:08.682042 6300 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
W20260620 19:59:08.458164 6904 consensus_meta.cc:287] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.740s user 0.006s sys 0.003s
W20260620 19:59:08.522171 8452 consensus_meta.cc:287] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.816s user 0.005s sys 0.002s
W20260620 19:59:08.662643 8638 consensus_meta.cc:287] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.984s user 0.010s sys 0.000s
I20260620 19:59:08.727646 6226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 371) took 1175 ms. Trace:
I20260620 19:59:08.512187 8736 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:08.726121 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:08.577476 8729 log.cc:927] Time spent T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
W20260620 19:59:08.724990 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 444929us
I20260620 19:59:08.516160 8643 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:08.662894 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.735344 6909 raft_consensus.cc:2410] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:08.508128 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 461) took 1244 ms. Trace:
I20260620 19:59:08.736078 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.423174 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:08.758253 8750 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:08.758253 8683 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:08.748176 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:08.773932 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:08.728045 6226 rpcz_store.cc:276] 0620 19:59:07.551896 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.667167 (+115271us) service_pool.cc:224] Handling call
0620 19:59:07.667472 (+ 305us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:07.910968 (+243496us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:07.910996 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:07.922218 (+ 11222us) log.cc:844] Serialized 52 byte log entry
0620 19:59:08.214691 (+292473us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:08.720768 (+506077us) log.cc:844] Serialized 14 byte log entry
0620 19:59:08.727308 (+ 6540us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:08.727381 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:08.727424 (+ 43us) raft_consensus.cc:1685] finished
0620 19:59:08.727435 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:08.727584 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":74935,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":123,"raft.queue_time_us":428283,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":33445,"thread_start_us":987192,"threads_started":3,"wal-append.queue_time_us":417814}
I20260620 19:59:08.729288 8712 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:08.744077 6304 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:08.758358 8721 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.773866 8683 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.715240 8730 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:08.807102 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:08.423234 8598 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.825407 8712 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:08.825448 6904 raft_consensus.cc:2410] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:08.743099 8630 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:08.758077 8726 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:08.863780 8712 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:08.864403 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1563) took 1306 ms. Trace:
I20260620 19:59:08.861088 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.866132 6378 rpcz_store.cc:276] 0620 19:59:07.558282 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.884491 (+326209us) service_pool.cc:224] Handling call
0620 19:59:08.172324 (+287833us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:08.283169 (+110845us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:08.864138 (+580969us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54000ec254. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:08.864334 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":253464960}
I20260620 19:59:08.394088 8740 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:08.862012 8743 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20260620 19:59:08.843700 6907 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:08.759305 6373 rpcz_store.cc:276] 0620 19:59:07.263685 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.305004 (+ 41319us) service_pool.cc:224] Handling call
0620 19:59:08.507901 (+1202897us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54000d4dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:08.508059 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":592142080}
I20260620 19:59:08.825706 6372 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:08.923367 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 796) took 1304 ms. Trace:
I20260620 19:59:08.864429 8711 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.932632 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.868067 8635 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:08.928102 6904 rpcz_store.cc:276] 0620 19:59:07.616979 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.661245 (+ 44266us) service_pool.cc:224] Handling call
0620 19:59:08.921664 (+1260419us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:08.929219 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.934635 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.945869 6300 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:08.730952 8728 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:08.937448 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 204595us
I20260620 19:59:08.947358 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:08.948374 6271 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:08.957653 6372 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:08.958796 6273 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:08.868782 8630 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:08.970940 6226 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260620 19:59:08.982337 6195 leader_election.cc:336] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.831s (SENT)
W20260620 19:59:08.972172 8682 consensus_meta.cc:287] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.572s user 0.006s sys 0.003s
I20260620 19:59:08.974298 8542 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:08.974416 8443 ts_tablet_manager.cc:1434] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent starting tablet: real 1.348s user 0.015s sys 0.024s
I20260620 19:59:08.986728 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:09.011158 6380 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1560) took 1458 ms (client timeout 1834 ms). Trace:
I20260620 19:59:09.013216 6231 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 5
I20260620 19:59:08.990773 8740 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:08.991710 6273 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:08.992929 8751 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.994786 8728 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.996243 8721 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:08.997499 8750 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:08.999022 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 793) took 1382 ms. Trace:
I20260620 19:59:09.002585 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:08.986872 8708 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
W20260620 19:59:09.011229 6381 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 1910) took 1852 ms (client timeout 1541 ms). Trace:
W20260620 19:59:09.013278 6380 rpcz_store.cc:269] 0620 19:59:07.552896 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.718929 (+166033us) service_pool.cc:224] Handling call
0620 19:59:09.003532 (+1284603us) spinlock_profiling.cc:243] Waited 612 ms on lock 0x7b5400063954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:09.011106 (+ 7574us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1408487808}
I20260620 19:59:09.016693 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:08.947954 8635 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.018110 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.068008 6227 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:08.984244 6298 raft_consensus.cc:2410] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:08.964365 8711 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260620 19:59:09.020524 6877 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.022837 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:09.071748 6907 raft_consensus.cc:2393] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:09.071897 6226 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:09.069500 6381 rpcz_store.cc:269] 0620 19:59:07.158845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.158953 (+ 108us) service_pool.cc:224] Handling call
0620 19:59:09.011066 (+1852113us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54000ea954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:09.011165 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1388859264}
W20260620 19:59:09.011351 8654 consensus_meta.cc:287] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.742s user 0.011s sys 0.000s
I20260620 19:59:09.055059 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.074761 6378 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:09.063604 6273 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:09.075810 8654 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:09.078647 8708 raft_consensus.cc:697] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:09.079633 8708 consensus_queue.cc:237] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.071631 8704 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.083069 8750 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.083132 8704 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.083698 8750 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.085448 6373 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:09.085800 8750 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:09.022436 8598 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260620 19:59:09.064282 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:09.086630 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.087097 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.087667 6907 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.067752 6905 rpcz_store.cc:276] 0620 19:59:07.616175 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.828501 (+212326us) service_pool.cc:224] Handling call
0620 19:59:07.930743 (+102242us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:08.256491 (+325748us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:08.998782 (+742291us) spinlock_profiling.cc:243] Waited 92.3 ms on lock 0x7b540068e2d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:08.998964 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":212180736}
I20260620 19:59:09.088732 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:09.089298 6905 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:09.090174 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.094105 8722 raft_consensus.cc:3055] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:09.026139 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:09.068934 6195 leader_election.cc:336] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.920s (SENT)
I20260620 19:59:09.024235 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.087214 6903 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.070952 6910 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:09.012954 8443 ts_tablet_manager.cc:1467] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Tablet startup took 1388ms
I20260620 19:59:09.108544 8345 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.113790 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.065328 8751 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.067649 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.054513 8737 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
I20260620 19:59:09.095757 6273 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:09.091779 8542 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.082160 8712 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:09.116446 6377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1559) took 1563 ms (client timeout 1831 ms). Trace:
W20260620 19:59:09.111949 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.095670 6907 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:09.075362 8730 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:09.019353 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 81227us
W20260620 19:59:09.120697 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:09.120604 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 51691.7us
I20260620 19:59:09.011776 8682 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.186496 8708 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.123224 8643 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.130724 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.135702 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 651) took 1859 ms. Trace:
I20260620 19:59:09.133231 8542 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.207060 8542 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.126590 6372 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:09.143652 8740 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:09.144325 6273 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:09.188611 8682 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:09.123612 8443 ts_tablet_manager.cc:1469] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Trace:
0620 19:58:57.383126 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:57.600304 (+217178us) service_pool.cc:224] Handling call
0620 19:58:58.275840 (+675536us) ts_tablet_manager.cc:934] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Initiating tablet copy from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
0620 19:59:00.566551 (+2290711us) tablet_copy_client.cc:368] Tablet copy session begun
0620 19:59:01.205666 (+639115us) tablet_metadata.cc:665] Metadata flushed
0620 19:59:01.205787 (+ 121us) tablet_copy_client.cc:497] Wrote new tablet metadata
0620 19:59:01.635162 (+429375us) tablet_copy_client.cc:418] Wrote new consensus metadata
0620 19:59:01.676718 (+ 41556us) tablet_replica.cc:206] Creating consensus instance
0620 19:59:01.686077 (+ 9359us) ts_tablet_manager.cc:989] Replica registered.
0620 19:59:01.686141 (+ 64us) inbound_call.cc:177] Queueing success response
0620 19:59:05.025486 (+3339345us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0620 19:59:07.624969 (+2599483us) ts_tablet_manager.cc:1435] Starting tablet replica
0620 19:59:07.707270 (+ 82301us) tablet_replica.cc:261] Starting instrumentation
0620 19:59:07.708024 (+ 754us) tablet_replica.cc:280] Starting consensus
0620 19:59:09.118936 (+1410912us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b1800092500. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b574340a 00007fa6bee885d9 00007fa6bc791a6f 00007fa6b57c8d71 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd
Metrics: {"dns_us":168398,"mutex_wait_us":22785,"raft.queue_time_us":63878,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":11580,"spinlock_wait_cycles":51607424,"tablet-copy.queue_time_us":674289,"tablet-download-pool.queue_time_us":176,"tablet-download-pool.run_cpu_time_us":1351,"tablet-download-pool.run_wall_time_us":2347872,"thread_start_us":671419,"threads_started":3}
I20260620 19:59:09.091073 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260620 19:59:09.186146 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 908325us
I20260620 19:59:09.193317 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:09.134323 6905 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:09.193791 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:09.194780 6875 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.194906 8691 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:09.119980 8722 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:09.197254 6345 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:09.228253 6230 raft_consensus.cc:2393] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:09.198172 8764 log.cc:927] Time spent T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260620 19:59:09.194916 8730 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.199395 8726 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.199536 8630 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:09.200454 8765 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
W20260620 19:59:09.202138 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:09.249372 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.121587 6231 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:09.252326 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 602) took 1325 ms. Trace:
I20260620 19:59:09.252691 6306 rpcz_store.cc:276] 0620 19:59:07.926840 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.927435 (+ 595us) service_pool.cc:224] Handling call
0620 19:59:07.927821 (+ 386us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:08.200828 (+273007us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:08.348530 (+147702us) log.cc:844] Serialized 14 byte log entry
0620 19:59:08.972902 (+624372us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:08.980764 (+ 7862us) log.cc:844] Serialized 26 byte log entry
0620 19:59:08.980913 (+ 149us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:08.980945 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:08.981002 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:09.252132 (+271130us) raft_consensus.cc:1685] finished
0620 19:59:09.252152 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:09.252273 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7411,"prepare.queue_time_us":414259,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":349861,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":31923,"spinlock_wait_cycles":49161344,"thread_start_us":690491,"threads_started":3,"wal-append.queue_time_us":567376}
I20260620 19:59:09.255419 6373 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:09.256632 8712 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.256923 6378 raft_consensus.cc:1240] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:59:09.257905 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:09.258926 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:09.259617 6380 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:09.151310 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.145623 6904 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:09.148310 6910 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
W20260620 19:59:09.202750 6377 rpcz_store.cc:269] 0620 19:59:07.552618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.688272 (+135654us) service_pool.cc:224] Handling call
0620 19:59:09.116252 (+1427980us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54002c9cd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:09.116380 (+ 128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":378115840}
I20260620 19:59:09.205433 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:09.205967 8643 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.215842 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.272666 6907 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:09.217806 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:09.220454 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 86657.7us
W20260620 19:59:09.261813 6344 leader_election.cc:336] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.035s (SENT)
I20260620 19:59:09.274559 6344 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:09.224984 8750 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260620 19:59:09.276602 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 256081us
I20260620 19:59:09.277335 6877 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.127044 8751 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.223580 6196 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:09.193400 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.226265 6875 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:09.206907 6902 rpcz_store.cc:276] 0620 19:59:07.276375 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.782582 (+506207us) service_pool.cc:224] Handling call
0620 19:59:07.783015 (+ 433us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:07.783293 (+ 278us) raft_consensus.cc:1534] Early marking committed up to index 7
0620 19:59:08.692061 (+908768us) log.cc:844] Serialized 14 byte log entry
0620 19:59:09.135208 (+443147us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:09.135292 (+ 84us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:09.135366 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:09.135457 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:09.135623 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":269497,"prepare.queue_time_us":970642,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":275385,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":1057691,"threads_started":2,"wal-append.queue_time_us":319776}
I20260620 19:59:09.229878 6345 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:09.213021 8708 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.228540 8638 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:09.233845 8737 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:09.240494 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:09.244108 8728 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:09.247605 8452 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:59:09.248555 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:09.297649 6379 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:59:09.258443 6381 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.298018 6908 raft_consensus.cc:2410] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260620 19:59:09.261341 6195 leader_election.cc:336] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.075s (SENT)
W20260620 19:59:09.284583 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.298043 6228 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
W20260620 19:59:09.302142 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 72309us
I20260620 19:59:09.307638 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.301203 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.310184 6378 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:09.314301 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.313601 6907 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:09.317572 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.295225 8750 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.312013 8345 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.010s
I20260620 19:59:09.295893 8751 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.295300 8740 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.301414 8737 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.297114 8474 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:09.296061 8708 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.324451 6376 raft_consensus.cc:1240] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260620 19:59:09.306862 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.301234 8711 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:09.327602 6874 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:09.329365 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.325133 6877 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.323360 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.352370 6381 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:09.352689 6378 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:09.300359 8728 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:09.329097 8776 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.319149 8753 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:09.314805 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 649) took 2037 ms. Trace:
I20260620 19:59:09.359825 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.312590 8763 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260620 19:59:09.349189 6909 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 639) took 2143 ms (client timeout 2677 ms). Trace:
W20260620 19:59:09.332022 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 105803us
W20260620 19:59:09.360961 6344 leader_election.cc:336] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.521s (SENT)
I20260620 19:59:09.361465 6344 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:09.323064 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 133420us
I20260620 19:59:09.326145 8750 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:09.361768 8763 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.399175 8763 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:09.361956 6906 rpcz_store.cc:276] 0620 19:59:07.274206 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.770175 (+495969us) service_pool.cc:224] Handling call
0620 19:59:07.790887 (+ 20712us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:08.198114 (+407227us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:08.200998 (+ 2884us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:08.304339 (+103341us) log.cc:844] Serialized 26 byte log entry
0620 19:59:08.935544 (+631205us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:08.963712 (+ 28168us) log.cc:844] Serialized 14 byte log entry
0620 19:59:09.294780 (+331068us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:09.294919 (+ 139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:09.294982 (+ 63us) raft_consensus.cc:1685] finished
0620 19:59:09.295014 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:09.311432 (+ 16418us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":187318,"prepare.queue_time_us":17941,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":441151,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":22836,"thread_start_us":818559,"threads_started":3,"wal-append.queue_time_us":630399}
I20260620 19:59:09.397382 6903 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:09.329337 8737 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:09.412300 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.367543 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:09.418956 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 93889.3us
I20260620 19:59:09.411849 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:09.418411 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 90839.7us
I20260620 19:59:09.421545 6307 raft_consensus.cc:2410] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260620 19:59:09.360988 6909 rpcz_store.cc:269] 0620 19:59:07.206134 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:07.561666 (+355532us) service_pool.cc:224] Handling call
0620 19:59:09.348991 (+1787325us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400564d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:09.349132 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2665573760}
I20260620 19:59:09.422912 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.423517 6379 raft_consensus.cc:1240] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:09.423661 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:09.427695 6299 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:09.423916 6877 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.425065 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.431330 6302 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:09.432391 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.432979 6302 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:09.433195 8704 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:09.433854 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:09.406987 8452 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:09.425189 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.350520 8474 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.435981 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.437075 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.437350 8474 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:09.437959 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.425623 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.439574 6303 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:09.440409 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.440908 6303 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:09.441604 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.426184 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.443070 6301 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:09.359690 8776 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:09.443647 8776 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.443866 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.444490 6301 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:09.445230 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:09.445703 6301 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:09.396397 8643 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:09.449095 8542 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.442513 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:09.450501 6303 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.452483 6198 leader_election.cc:400] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5.
I20260620 19:59:09.452805 6198 leader_election.cc:403] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:59:09.453190 6905 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:09.453986 6300 raft_consensus.cc:2393] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:09.454856 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.455345 6300 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:09.456231 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.456559 6874 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.456959 8735 consensus_queue.cc:1059] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:09.434405 6302 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:09.457867 8436 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.458364 8436 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.458550 8682 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:09.459921 8682 consensus_queue.cc:1059] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:09.460345 8436 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:09.461047 8436 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.461514 8436 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.455297 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:09.424572 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.420652 6373 raft_consensus.cc:1275] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:59:09.463559 6306 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:09.466121 6874 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.464104 6304 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 2.
I20260620 19:59:09.450304 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.465031 6198 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:09.361498 6875 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.472127 6303 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:09.330716 8740 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.442387 6904 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:09.359993 8728 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:09.476467 6380 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:09.481251 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 257720us
W20260620 19:59:09.476738 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 173191us
I20260620 19:59:09.476303 6301 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:09.476441 6299 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:09.480360 6874 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:09.481531 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 61661us
I20260620 19:59:09.481993 6196 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:09.457794 8735 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5.
I20260620 19:59:09.483728 6877 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:09.482829 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 183925us
W20260620 19:59:09.486299 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.320317 8722 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.489449 6877 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.476854 8726 raft_consensus.cc:3055] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:09.483847 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:09.476511 8691 raft_consensus.cc:3055] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:09.556416 8721 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260620 19:59:09.555537 8542 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.580387 8436 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:09.555639 8735 consensus_queue.cc:1048] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260620 19:59:09.555217 8683 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:09.585705 8683 raft_consensus.cc:697] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:09.586659 8683 consensus_queue.cc:237] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:09.564354 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 80682.5us
I20260620 19:59:09.508966 6306 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:09.494156 8726 raft_consensus.cc:740] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:09.530189 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 65202.3us
I20260620 19:59:09.591146 6198 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:09.594046 8777 log.cc:927] Time spent T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.199s user 0.004s sys 0.000s
I20260620 19:59:09.595959 8763 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.598230 8763 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:09.497788 6130 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45406 (request call id 267) took 12532 ms (client timeout 15000 ms). Trace:
I20260620 19:59:09.484723 8474 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.521554 6196 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:09.681819 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 160327us
I20260620 19:59:09.683949 6196 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:09.486135 8618 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:09.554433 8638 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:09.494966 8643 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:09.491885 8654 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:09.524474 6302 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:09.497692 8760 consensus_queue.cc:1059] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260620 19:59:09.477020 8775 raft_consensus.cc:2804] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:59:09.477172 8738 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260620 19:59:09.504745 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:09.489275 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:09.712196 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 222900us
I20260620 19:59:09.713351 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:09.528065 8574 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.564908 8606 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:09.567066 8776 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.573767 8740 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:09.575805 8708 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.578752 8691 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:09.598770 8750 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:09.601714 8635 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:09.603394 8751 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.608651 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:09.610323 6228 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:59:09.610898 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:09.611789 6377 raft_consensus.cc:1240] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:09.609236 8726 consensus_queue.cc:260] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.612890 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:09.613849 6231 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
W20260620 19:59:09.619809 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.715962 8574 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.716306 8436 raft_consensus.cc:2764] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260620 19:59:09.563249 8781 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:09.504251 8722 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:09.540853 6195 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:09.666069 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 173259us
I20260620 19:59:09.765678 6877 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.767966 6877 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.768961 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:09.482617 8756 raft_consensus.cc:2764] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260620 19:59:09.633167 8735 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:09.770407 8735 raft_consensus.cc:697] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:09.771494 8735 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.708163 6128 catalog_manager.cc:5696] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 4 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:59:09.776300 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 295913us
I20260620 19:59:09.775830 6128 catalog_manager.cc:5696] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:09.777763 6874 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:09.709100 8737 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.780229 8737 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:09.715080 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:09.781036 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 66480.2us
I20260620 19:59:09.781677 6271 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:09.699342 8778 log.cc:927] Time spent T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.269s user 0.002s sys 0.000s
I20260620 19:59:09.680357 8712 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.784170 8712 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:09.687741 8474 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.712134 6196 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:09.691026 8721 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:09.786846 8474 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.787067 8721 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:09.787245 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 76231.5us
W20260620 19:59:09.718209 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 300314us
I20260620 19:59:09.788141 6196 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:09.789665 8746 raft_consensus.cc:2804] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:09.789774 6196 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:09.790057 8746 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:09.790405 8746 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:09.702342 8643 raft_consensus.cc:697] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:09.790759 8618 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:09.791507 8618 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:09.791963 8643 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.793123 8618 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:09.710446 8775 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:09.794822 8775 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:09.732820 6380 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:09.799090 8712 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:59:09.778184 6128 catalog_manager.cc:5696] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 4, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 4 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:09.718223 8542 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260620 19:59:09.740469 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.802454 8542 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:09.802304 8746 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.802987 8542 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:59:09.619946 8774 log.cc:927] Time spent T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.194s user 0.002s sys 0.000s
I20260620 19:59:09.804497 6381 raft_consensus.cc:1240] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:09.804908 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.805480 6377 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:09.805546 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:09.806246 6374 raft_consensus.cc:2393] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:09.742012 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.787518 8721 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:09.806437 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:09.807166 6379 raft_consensus.cc:2393] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:09.719920 8654 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260620 19:59:09.802003 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:09.808216 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.808207 8654 raft_consensus.cc:2764] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260620 19:59:09.807809 6128 catalog_manager.cc:5696] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 5, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:09.809741 8746 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.690876 8730 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.810796 8730 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:09.811203 8730 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.811602 8618 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:09.819459 8775 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.816781 6380 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:09.818174 8730 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.821579 8721 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.825278 8730 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.821823 8775 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:09.824066 8721 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:09.827960 8775 raft_consensus.cc:2764] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260620 19:59:09.827617 6271 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:09.828848 8775 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:09.829438 8775 raft_consensus.cc:697] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:09.717000 6298 raft_consensus.cc:2376] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7. Re-sending same reply.
I20260620 19:59:09.724424 8606 raft_consensus.cc:697] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:09.835795 8606 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.719717 8638 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260620 19:59:09.832683 6305 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:09.732597 8740 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:09.809399 6128 catalog_manager.cc:5696] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 4, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:59:09.834566 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:09.839531 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 298669us
I20260620 19:59:09.841733 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:09.843353 6195 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:09.844959 6128 catalog_manager.cc:5696] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:59:09.847792 8691 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.848289 8787 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:59:09.849258 8691 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:09.851879 8776 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:09.758903 8574 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.855770 8682 raft_consensus.cc:3055] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:09.856149 8682 raft_consensus.cc:740] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:09.856879 8682 consensus_queue.cc:260] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:09.858112 8682 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:09.859972 8750 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:59:09.733043 8728 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.861647 8750 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:09.862282 8728 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:09.864264 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:09.864681 6344 leader_election.cc:336] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.091s (SENT)
I20260620 19:59:09.866448 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:09.869307 6198 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:09.869953 6345 leader_election.cc:336] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 1.577s (SENT)
I20260620 19:59:09.870378 6345 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:09.873944 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 614) took 1360 ms. Trace:
I20260620 19:59:09.874498 6305 rpcz_store.cc:276] 0620 19:59:08.513707 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:08.577443 (+ 63736us) service_pool.cc:224] Handling call
0620 19:59:09.190644 (+613201us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b5800003098. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:59:09.873711 (+683067us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54003a6954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:09.873874 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1326025472}
I20260620 19:59:09.877954 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:09.879226 6195 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:09.918639 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 557033us
W20260620 19:59:09.943320 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.919726 6128 catalog_manager.cc:5696] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } }
W20260620 19:59:09.670027 6130 rpcz_store.cc:269] 0620 19:58:56.940568 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:58:56.940826 (+ 258us) service_pool.cc:224] Handling call
0620 19:59:09.466819 (+12525993us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:59:03.810821 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:59:03.811602 (+ 781us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:59:03.811648 (+ 46us) write_op.cc:435] Acquired schema lock
0620 19:59:03.811675 (+ 27us) tablet.cc:662] Decoding operations
0620 19:59:03.831144 (+ 19469us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:59:03.831227 (+ 83us) write_op.cc:641] Partition lock acquired for write op
0620 19:59:03.831254 (+ 27us) tablet.cc:685] Acquiring locks for 6 operations
0620 19:59:03.831745 (+ 491us) tablet.cc:701] Row locks acquired
0620 19:59:03.831763 (+ 18us) write_op.cc:260] PREPARE: finished
0620 19:59:03.831904 (+ 141us) write_op.cc:270] Start()
0620 19:59:03.832015 (+ 111us) write_op.cc:276] Timestamp: P: 1781985543831856 usec, L: 0
0620 19:59:03.832039 (+ 24us) op_driver.cc:348] REPLICATION: starting
0620 19:59:03.842165 (+ 10126us) log.cc:844] Serialized 2452 byte log entry
0620 19:59:04.743726 (+901561us) op_driver.cc:464] REPLICATION: finished
0620 19:59:05.001430 (+257704us) write_op.cc:301] APPLY: starting
0620 19:59:05.043044 (+ 41614us) tablet.cc:1367] starting BulkCheckPresence
0620 19:59:05.053080 (+ 10036us) tablet.cc:1370] finished BulkCheckPresence
0620 19:59:05.053231 (+ 151us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:59:05.121268 (+ 68037us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:59:05.816913 (+695645us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0620 19:59:05.822005 (+ 5092us) write_op.cc:312] APPLY: finished
0620 19:59:05.870968 (+ 48963us) log.cc:844] Serialized 53 byte log entry
0620 19:59:06.143874 (+272906us) spinlock_profiling.cc:243] Waited 7.5 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b57ab8ac 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab
0620 19:59:06.258391 (+114517us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:06.258684 (+ 293us) write_op.cc:454] Released schema lock
0620 19:59:06.258762 (+ 78us) write_op.cc:341] FINISH: Updating metrics
0620 19:59:06.259445 (+ 683us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:06.259475 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":951072,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":247823,"apply.run_cpu_time_us":81785,"apply.run_wall_time_us":1261899,"num_ops":6,"prepare.queue_time_us":75713,"prepare.run_cpu_time_us":16600,"prepare.run_wall_time_us":299902,"raft.queue_time_us":246064,"raft.run_cpu_time_us":13082,"raft.run_wall_time_us":94350,"replication_time_us":911426,"spinlock_wait_cycles":17246592,"thread_start_us":1046809,"threads_started":5,"wal-append.queue_time_us":802224,"wal-append.run_cpu_time_us":52555,"wal-append.run_wall_time_us":3215356}]]}
I20260620 19:59:09.880558 8618 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:59:09.912165 8730 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:09.840231 8783 raft_consensus.cc:2764] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260620 19:59:09.973277 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:09.865621 8541 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.881786 8726 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:09.991855 8541 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.994155 8541 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:09.975972 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 206992us
I20260620 19:59:09.977669 6875 leader_election.cc:400] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7.
I20260620 19:59:09.997673 6875 leader_election.cc:403] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:59:09.862991 8740 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:09.979704 6273 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:09.837325 8775 consensus_queue.cc:237] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.847903 8683 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:09.881347 8733 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:09.972466 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 102506us
I20260620 19:59:09.880955 8760 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:09.882048 8721 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:09.942689 8746 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:09.998453 6877 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:09.974833 8711 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:10.003784 6128 catalog_manager.cc:5696] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 3, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:59:10.005141 8754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.013s sys 0.030s
I20260620 19:59:10.021931 6128 catalog_manager.cc:5696] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } }
I20260620 19:59:09.990578 8753 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:09.981465 8583 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.831349 8781 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:09.984424 8618 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:10.003197 8740 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:10.007195 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:09.972271 8542 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.028700 6877 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:10.030905 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:10.031730 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 253982us
I20260620 19:59:10.006067 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:10.013370 8735 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:10.037959 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:10.035494 6875 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:10.019486 8606 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:09.983484 8436 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.039s
I20260620 19:59:10.007157 8728 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260620 19:59:10.025925 8754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.013s sys 0.030s
I20260620 19:59:10.040201 6907 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:10.023975 6128 catalog_manager.cc:5696] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: config changed from index -1 to 3, term changed from 2 to 4, NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) added. New cstate: current_term: 4 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:59:10.025830 8794 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 7
W20260620 19:59:10.038785 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 60382.8us
W20260620 19:59:10.050345 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 171169us
I20260620 19:59:10.114286 6195 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:10.026198 8721 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:10.028767 8618 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
W20260620 19:59:10.097069 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 66207.8us
I20260620 19:59:10.102422 8711 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:10.016840 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 227062us
I20260620 19:59:10.126385 6196 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:10.104414 8728 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:10.105260 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 64882.7us
I20260620 19:59:10.128603 6874 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:10.107685 6128 catalog_manager.cc:5696] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 5, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 5 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } }
W20260620 19:59:10.129657 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:10.115841 6195 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:10.020445 8746 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:10.119037 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 118230us
I20260620 19:59:10.130157 6128 catalog_manager.cc:5696] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 4, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:10.131368 6273 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:10.099956 8756 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:10.122274 6345 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:10.116434 8737 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:10.141383 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 115635us
I20260620 19:59:10.038400 8776 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:10.006304 8786 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260620 19:59:10.019716 8787 consensus_queue.cc:1059] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
I20260620 19:59:10.040378 8735 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:10.111091 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:10.132862 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:10.114084 8785 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:10.026057 8760 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.119113 8643 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:10.143450 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:10.143700 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:10.191062 6379 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:10.140641 8756 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:10.155467 8541 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:10.145341 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:10.146341 6195 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:10.195989 6301 raft_consensus.cc:2368] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6.
W20260620 19:59:10.196211 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 49970.4us
I20260620 19:59:10.197817 6195 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:10.148288 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:10.037175 8801 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.102s
I20260620 19:59:10.037246 8583 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.121039 8779 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:10.163399 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:10.132519 8746 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:10.171300 6344 leader_election.cc:336] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.798s (SENT)
I20260620 19:59:10.165066 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:10.169852 6128 catalog_manager.cc:5696] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } }
I20260620 19:59:10.123682 8799 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:10.123375 8751 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:10.176367 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:10.135258 8728 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:10.165143 8737 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:10.105496 8798 consensus_queue.cc:1059] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260620 19:59:10.106298 8606 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:10.144408 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:10.192435 6306 raft_consensus.cc:2410] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260620 19:59:10.195122 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 62346.7us
W20260620 19:59:10.198783 6196 connection.cc:256] RPC call timeout handler was delayed by -1.4904996559999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2677s for a response
I20260620 19:59:10.157256 8776 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:10.190590 8643 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:10.175019 8735 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
W20260620 19:59:10.259232 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 61468.5us
I20260620 19:59:10.359688 6195 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:10.280774 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 109481us
I20260620 19:59:10.189863 8750 consensus_queue.cc:1059] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.100s
W20260620 19:59:10.240931 8803 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260620 19:59:10.199407 8730 raft_consensus.cc:2764] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260620 19:59:10.357107 6345 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:10.247452 8804 log.cc:927] Time spent T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260620 19:59:10.344178 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:10.246939 8788 log.cc:927] Time spent T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
I20260620 19:59:10.281293 6380 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:10.243456 8784 log.cc:927] Time spent T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.260s user 0.000s sys 0.001s
I20260620 19:59:10.303323 8738 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:10.242465 8792 log.cc:927] Time spent T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20260620 19:59:10.241551 8791 log.cc:927] Time spent T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.181s user 0.000s sys 0.000s
W20260620 19:59:10.242995 8793 log.cc:927] Time spent T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.180s user 0.000s sys 0.000s
I20260620 19:59:10.372341 6195 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:10.384693 6299 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:10.203504 8756 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:10.195714 8541 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:10.242869 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:10.205235 8711 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:10.289697 8802 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:59:10.364207 8643 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:10.373067 8574 consensus_meta.cc:287] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.519s user 0.008s sys 0.003s
W20260620 19:59:10.389662 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:10.360714 8776 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:10.202800 8806 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:10.378799 8751 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:10.363898 8754 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:10.411788 6195 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:10.278571 8683 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:10.247160 8474 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.298789 8785 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:10.192826 8790 consensus_queue.cc:1059] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.413s
I20260620 19:59:10.380383 8712 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:10.385524 8808 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:10.298028 8618 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:10.280037 8721 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:10.437129 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:10.420159 8541 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:10.419780 8799 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
W20260620 19:59:10.478853 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 336723us
W20260620 19:59:10.495092 6271 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.013s sys 0.017s
W20260620 19:59:10.500892 6271 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s user 0.013s sys 0.017s
I20260620 19:59:10.462651 6380 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
W20260620 19:59:10.429215 8682 consensus_meta.cc:287] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.571s user 0.008s sys 0.003s
I20260620 19:59:10.427831 8802 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:10.394567 8738 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.488458 8811 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:10.440685 8574 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.498373 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 819) took 1479 ms. Trace:
I20260620 19:59:10.533210 6911 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:10.437124 8781 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:10.429760 8726 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:10.441517 8798 raft_consensus.cc:3055] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Stepping down as leader of term 5
W20260620 19:59:10.505558 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 677809us
I20260620 19:59:10.507477 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:10.557770 6228 raft_consensus.cc:2368] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6.
I20260620 19:59:10.508539 6230 raft_consensus.cc:1240] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:59:10.484166 8809 raft_consensus.cc:3055] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:10.432305 8542 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:10.525960 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:10.497000 8806 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:10.481503 8751 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:10.558112 6906 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:10.533838 6908 raft_consensus.cc:1240] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:10.526566 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:10.534101 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 222691us
W20260620 19:59:10.534550 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 123291us
I20260620 19:59:10.574615 6231 raft_consensus.cc:2368] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
W20260620 19:59:10.534816 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:10.504055 8808 raft_consensus.cc:697] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:10.532475 6910 rpcz_store.cc:276] 0620 19:59:09.019163 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:09.271511 (+252348us) service_pool.cc:224] Handling call
0620 19:59:09.271809 (+ 298us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:09.312255 (+ 40446us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:09.312294 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:09.326508 (+ 14214us) log.cc:844] Serialized 26 byte log entry
0620 19:59:09.694523 (+368015us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:09.985557 (+291034us) log.cc:844] Serialized 14 byte log entry
0620 19:59:09.986108 (+ 551us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:09.986515 (+ 407us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:10.498181 (+511666us) raft_consensus.cc:1685] finished
0620 19:59:10.498201 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:10.498319 (+ 118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42770,"prepare.queue_time_us":508039,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":39412,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":23638,"thread_start_us":586553,"threads_started":2,"wal-append.queue_time_us":412917}
I20260620 19:59:10.511698 8799 raft_consensus.cc:697] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:10.559175 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:10.579841 6911 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:10.559634 6875 leader_election.cc:400] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6.
I20260620 19:59:10.513839 8802 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:10.494047 8785 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:10.534857 8722 consensus_meta.cc:287] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.702s user 0.000s sys 0.009s
I20260620 19:59:10.567813 6232 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
W20260620 19:59:10.495949 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 84243.8us
I20260620 19:59:10.573347 6271 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:10.580305 6875 leader_election.cc:403] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260620 19:59:10.558619 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 180152us
I20260620 19:59:10.588178 6345 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:10.535068 8740 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.563515 8809 raft_consensus.cc:740] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:10.587208 6195 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:10.557260 8798 raft_consensus.cc:740] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:10.576483 8751 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:10.566272 8726 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:10.581754 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:10.614877 6271 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:10.589491 6345 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:10.531912 8811 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.587248 8737 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:10.583209 8583 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:10.638522 8726 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:10.639102 6874 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:10.639845 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:10.610118 8606 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:10.644330 8775 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:10.631325 8783 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:10.644902 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 55457.4us
W20260620 19:59:10.686484 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 71662.2us
I20260620 19:59:10.691427 6875 leader_election.cc:400] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
I20260620 19:59:10.691877 6875 leader_election.cc:403] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260620 19:59:10.612866 8781 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:10.692317 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:10.684799 6345 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:10.694839 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:10.696975 6231 raft_consensus.cc:2410] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260620 19:59:10.698738 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 59774.4us
I20260620 19:59:10.700966 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:10.703193 6345 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:10.704105 6345 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:10.709686 6232 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260620 19:59:10.710003 8820 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:10.710418 8820 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:10.710939 8474 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:10.711687 8474 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:10.712360 8474 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:10.639439 8790 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:10.713646 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:10.714234 6380 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:10.715229 8794 consensus_meta.cc:287] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.590s user 0.009s sys 0.000s
I20260620 19:59:10.715700 8794 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7.
I20260620 19:59:10.716741 8574 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:10.719115 6198 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:10.719642 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:10.710774 8820 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:10.720947 6273 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:10.588071 8708 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:10.588883 8812 consensus_queue.cc:1059] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.097s
I20260620 19:59:10.686218 8821 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
W20260620 19:59:10.733031 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 45287.6us
I20260620 19:59:10.718417 8606 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:10.730581 8737 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:10.586289 8813 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.148s
I20260620 19:59:10.731892 8711 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:10.729898 6911 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:10.747951 8691 consensus_meta.cc:287] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.898s user 0.011s sys 0.001s
I20260620 19:59:10.759961 6271 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:10.729131 8806 raft_consensus.cc:697] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:10.763753 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:10.769559 6911 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:10.760488 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:10.748144 8726 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:10.683794 8783 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.773533 6380 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260620 19:59:10.776264 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:10.761831 8737 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:10.777572 6378 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260620 19:59:10.777719 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 86434.4us
I20260620 19:59:10.769160 8606 raft_consensus.cc:697] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:10.761524 8808 consensus_queue.cc:237] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:10.781136 6375 raft_consensus.cc:2393] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:10.891381 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:10.893096 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:10.893242 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:10.893923 6373 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:10.895027 6390 tablet_copy_service.cc:342] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Request end of tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-42a982b7017a41e1b0ca419a82cacf59 received from {username='slave'} at 127.0.0.1:46186
I20260620 19:59:10.895591 6390 tablet_copy_service.cc:434] P 3a9a1e04ea8f425f8bcec4439ee5ebf9: ending tablet copy session 20bd146dc0cb4917a1a5ce05ec609ce1-42a982b7017a41e1b0ca419a82cacf59 on tablet 42a982b7017a41e1b0ca419a82cacf59 with peer 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:10.893946 6379 raft_consensus.cc:2393] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:10.898808 6375 raft_consensus.cc:1240] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:10.900559 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:10.901654 6911 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:10.945856 6345 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:10.951125 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:10.867990 8816 log.cc:927] Time spent T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
W20260620 19:59:10.962371 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 202454us
W20260620 19:59:10.866544 8810 log.cc:927] Time spent T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.298s user 0.000s sys 0.001s
I20260620 19:59:10.955683 6875 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:10.972929 6271 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:10.871402 8826 log.cc:927] Time spent T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260620 19:59:10.974227 6378 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:10.750991 8809 consensus_queue.cc:260] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:10.782024 8737 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:10.955760 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:10.866917 8829 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:10.738963 8799 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:10.780144 8738 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:10.867059 8817 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:10.979082 8583 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:10.865350 8726 raft_consensus.cc:697] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:10.997695 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:11.006124 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 50517.7us
I20260620 19:59:10.778256 8712 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:10.819352 8691 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.182s
I20260620 19:59:10.980892 8781 raft_consensus.cc:697] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:11.016098 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:10.999421 8829 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:10.867159 8818 consensus_queue.cc:1059] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.314s
I20260620 19:59:11.002929 8809 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:11.005263 8583 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:11.033514 8836 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.000916 8828 raft_consensus.cc:2764] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260620 19:59:11.034051 8726 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.014432 8712 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:11.007632 8821 raft_consensus.cc:697] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:10.979871 8740 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:11.038864 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 288083us
W20260620 19:59:11.071273 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 483971us
W20260620 19:59:11.011200 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 311257us
I20260620 19:59:11.050923 6232 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260620 19:59:11.037314 8812 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:11.056344 8838 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:11.002120 8825 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.000068 8824 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:11.059239 8583 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:11.035722 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:11.087914 6911 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:11.018760 8785 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:11.049633 8790 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.072480 6195 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:11.075565 6874 leader_election.cc:400] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6.
I20260620 19:59:11.100773 6345 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:11.072057 6196 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:11.102630 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:10.983255 8806 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.040709 8829 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:11.077780 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 106033us
I20260620 19:59:11.036666 8738 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260620 19:59:11.109555 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.128353 6307 raft_consensus.cc:2393] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:11.110755 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:11.077363 8818 consensus_queue.cc:1243] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260620 19:59:11.076740 8812 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:11.114972 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:11.120152 8833 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
W20260620 19:59:11.124832 8779 consensus_meta.cc:287] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.762s user 0.007s sys 0.002s
I20260620 19:59:11.075204 8811 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:11.126823 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:11.085801 8711 raft_consensus.cc:697] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:11.119791 6271 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:11.170440 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 50704.2us
I20260620 19:59:11.137843 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:11.171279 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:11.057612 8606 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.143796 6301 raft_consensus.cc:3060] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:11.150738 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:11.133947 8756 consensus_meta.cc:287] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.706s user 0.006s sys 0.004s
I20260620 19:59:11.171638 6230 raft_consensus.cc:2410] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260620 19:59:11.140830 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:11.188431 6307 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:11.178828 6229 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:11.133267 8775 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6.
I20260620 19:59:11.195732 6225 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260620 19:59:11.139925 8812 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:11.135684 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 824) took 1101 ms. Trace:
I20260620 19:59:11.088785 8825 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.135316 8783 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:11.169931 8779 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:11.220403 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:11.231276 6230 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:11.076231 8836 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:11.233196 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:11.116250 8790 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:11.198805 8828 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:11.245182 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:11.113328 8781 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.096812 8824 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.258066 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:11.272135 6230 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:11.234519 8817 raft_consensus.cc:697] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:11.276530 6303 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260620 19:59:11.199889 8756 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.250342 8783 raft_consensus.cc:2764] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260620 19:59:11.200388 8821 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.252460 8799 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:11.407714 8861 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.254731 6223 raft_consensus.cc:1240] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:11.448930 8846 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:11.452059 8846 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:11.452427 8846 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:11.312449 8738 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.455754 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:11.365250 8827 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260620 19:59:11.459959 6379 raft_consensus.cc:3060] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:11.462922 6380 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260620 19:59:11.382735 8837 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.355507 8841 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.467994 6911 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:11.365904 8712 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:11.474689 6301 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:11.477208 8852 log.cc:927] Time spent T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20260620 19:59:11.481441 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:11.470880 8738 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.484436 8841 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:11.485913 8738 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:11.471640 8861 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.475255 8833 raft_consensus.cc:697] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:11.487522 6374 raft_consensus.cc:1240] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260620 19:59:11.496958 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:11.500485 6381 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:11.493450 6307 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
W20260620 19:59:11.517057 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:11.478421 8859 log.cc:927] Time spent T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260620 19:59:11.548244 6907 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:11.546481 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.555667 6307 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:11.563910 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:11.281028 8711 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:11.490870 8751 consensus_meta.cc:287] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.861s user 0.013s sys 0.007s
I20260620 19:59:11.597350 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:11.625398 8850 log.cc:927] Time spent T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.247s user 0.001s sys 0.000s
I20260620 19:59:11.488719 8861 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:11.491005 8691 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.632360 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:59:11.488427 8836 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:11.486559 8841 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:11.453760 8799 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:11.211344 6904 rpcz_store.cc:276] 0620 19:59:10.034674 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.059517 (+ 24843us) service_pool.cc:224] Handling call
0620 19:59:10.496873 (+437356us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:10.508775 (+ 11902us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:10.534399 (+ 25624us) raft_consensus.cc:1534] Early marking committed up to index 10
0620 19:59:10.696979 (+162580us) log.cc:844] Serialized 14 byte log entry
0620 19:59:11.086417 (+389438us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:11.088765 (+ 2348us) raft_consensus.cc:1645] Marking committed up to 10
0620 19:59:11.099530 (+ 10765us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:11.116697 (+ 17167us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:11.133106 (+ 16409us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15855,"prepare.queue_time_us":31022,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":10951,"raft.queue_time_us":21765,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":10821,"spinlock_wait_cycles":6880000,"thread_start_us":397573,"threads_started":2,"wal-append.queue_time_us":469796}
W20260620 19:59:11.645936 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:11.654186 6231 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.017s sys 0.067s
W20260620 19:59:11.654878 6231 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s user 0.017s sys 0.067s
I20260620 19:59:11.633312 6380 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:11.663106 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.664072 6307 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:59:11.489144 8583 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.669420 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:11.670238 6307 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:11.562407 8783 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260620 19:59:11.660234 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:11.690248 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:11.635728 8751 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:11.694871 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:11.697444 6231 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
W20260620 19:59:11.698853 8854 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20260620 19:59:11.698935 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:11.699831 8618 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:11.702986 8855 log.cc:927] Time spent T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.314s user 0.001s sys 0.000s
I20260620 19:59:11.703058 8834 raft_consensus.cc:3055] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:11.703538 8834 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:11.705116 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 826) took 1669 ms. Trace:
W20260620 19:59:11.687815 8820 consensus_meta.cc:287] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.935s user 0.012s sys 0.009s
I20260620 19:59:11.705523 6909 rpcz_store.cc:276] 0620 19:59:10.036491 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.060398 (+ 23907us) service_pool.cc:224] Handling call
0620 19:59:10.300475 (+240077us) spinlock_profiling.cc:243] Waited 7.88 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:10.300671 (+ 196us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:10.321899 (+ 21228us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:10.321945 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:10.508594 (+186649us) spinlock_profiling.cc:243] Waited 8.55 ms on lock 0x7b2c000125b8. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b71b3c71 00007fa6b71c35d2 00007fa6b88f5de1 00007fa6b88ba81a 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22
0620 19:59:10.733137 (+224543us) spinlock_profiling.cc:243] Waited 9.25 ms on lock 0x7b2c000125b8. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b71b3c71 00007fa6b71c35d2 00007fa6b88f5de1 00007fa6b88ba81a 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22
0620 19:59:10.751342 (+ 18205us) log.cc:844] Serialized 52 byte log entry
0620 19:59:11.190353 (+439011us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:11.208608 (+ 18255us) log.cc:844] Serialized 14 byte log entry
0620 19:59:11.208814 (+ 206us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:11.208883 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:11.704885 (+496002us) raft_consensus.cc:1685] finished
0620 19:59:11.704904 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:11.705042 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24064,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":4949,"raft.queue_time_us":117465,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":7434,"spinlock_wait_cycles":169269120,"thread_start_us":417982,"threads_started":1,"wal-append.queue_time_us":494210}
I20260620 19:59:11.716270 6301 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:11.682060 8738 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:11.727072 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.654757 8691 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:11.690562 8845 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20260620 19:59:11.723704 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:11.728459 6300 raft_consensus.cc:2368] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4.
I20260620 19:59:11.729981 6195 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:11.491811 8833 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.735194 6195 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:11.738018 6307 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:11.503990 8837 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.740139 6874 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:11.742197 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.725488 8836 raft_consensus.cc:3055] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:11.548596 8817 consensus_queue.cc:237] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:11.733680 8842 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:11.750845 6228 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:11.744186 6307 raft_consensus.cc:2393] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:11.749555 8790 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:11.791047 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:11.793964 6301 raft_consensus.cc:2376] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6. Re-sending same reply.
I20260620 19:59:11.796445 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:11.797363 6377 raft_consensus.cc:1240] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
W20260620 19:59:11.801322 6229 consensus_meta.cc:287] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.612s user 0.018s sys 0.005s
I20260620 19:59:11.801762 6229 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 9.
I20260620 19:59:11.749584 8756 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:11.740170 8841 raft_consensus.cc:2804] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:11.743724 8862 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
W20260620 19:59:11.750682 8728 consensus_meta.cc:287] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 1.380s user 0.012s sys 0.008s
W20260620 19:59:11.751315 8643 consensus_meta.cc:287] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 1.311s user 0.010s sys 0.002s
I20260620 19:59:11.704423 8834 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:11.800580 8746 consensus_meta.cc:287] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.422s user 0.018s sys 0.011s
I20260620 19:59:11.744761 8740 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:11.799240 8753 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:11.820415 6232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.018s sys 0.040s
W20260620 19:59:11.796475 8735 consensus_meta.cc:287] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.420s user 0.012s sys 0.000s
W20260620 19:59:11.753894 8856 log.cc:927] Time spent T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20260620 19:59:11.744882 8836 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:11.733038 8738 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.837598 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.913954 6299 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:11.751268 8824 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:11.729687 8820 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:11.901219 6232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s user 0.025s sys 0.047s
I20260620 19:59:11.829113 8841 raft_consensus.cc:697] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:11.838912 8862 raft_consensus.cc:697] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:11.838980 8728 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.840330 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 658) took 1308 ms. Trace:
I20260620 19:59:11.863232 8643 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:11.957089 6307 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
W20260620 19:59:12.036164 6874 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.900s (SENT) [suppressed 55 similar messages]
W20260620 19:59:12.042721 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:12.062341 6379 consensus_meta.cc:287] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.597s user 0.007s sys 0.000s
I20260620 19:59:12.063241 6379 raft_consensus.cc:2468] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:12.053718 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.954208 6195 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:11.835834 8787 raft_consensus.cc:3055] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:11.937031 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:11.924883 8776 consensus_meta.cc:287] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 1.442s user 0.009s sys 0.003s
W20260620 19:59:11.909708 8474 consensus_meta.cc:287] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 1.197s user 0.007s sys 0.006s
I20260620 19:59:12.065397 6299 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:12.075105 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:12.065456 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 111333us
I20260620 19:59:11.923234 8834 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:12.053615 8683 consensus_meta.cc:287] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.569s user 0.002s sys 0.010s
I20260620 19:59:12.070686 6302 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:12.076107 6301 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 7
I20260620 19:59:11.910779 8746 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:11.927286 8740 raft_consensus.cc:697] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:12.026484 8838 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.063771 8756 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:12.106674 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:11.924645 8753 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.067314 8860 log.cc:927] Time spent T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.429s user 0.001s sys 0.000s
W20260620 19:59:12.090376 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.110107 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:11.950696 8842 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.040745 8583 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.071375 8871 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:11.901507 8846 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:12.161034 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 124878us
I20260620 19:59:12.068172 8787 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:12.067488 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 631) took 1347 ms. Trace:
I20260620 19:59:11.923579 8735 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:12.071058 8751 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.047573 8824 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:12.126157 6306 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.021s sys 0.045s
I20260620 19:59:12.128598 6225 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:12.080972 8474 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 3. Current term is 4.
I20260620 19:59:12.131104 6307 raft_consensus.cc:2368] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5.
W20260620 19:59:12.194990 6306 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.960s user 0.022s sys 0.045s
I20260620 19:59:12.196134 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.107788 8750 raft_consensus.cc:3055] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
W20260620 19:59:12.121443 8541 consensus_meta.cc:287] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 1.616s user 0.013s sys 0.000s
I20260620 19:59:12.113097 8683 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:12.111085 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:12.136180 8812 consensus_meta.cc:287] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.912s user 0.007s sys 0.003s
W20260620 19:59:12.137178 8872 log.cc:927] Time spent T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260620 19:59:12.141479 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1552) took 1035 ms. Trace:
I20260620 19:59:12.142043 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1549) took 1037 ms. Trace:
I20260620 19:59:12.143991 8643 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.215731 6224 raft_consensus.cc:1240] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:12.146319 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 476) took 1256 ms. Trace:
I20260620 19:59:12.216287 6375 rpcz_store.cc:276] 0620 19:59:10.889762 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.897170 (+ 7408us) service_pool.cc:224] Handling call
0620 19:59:10.898588 (+ 1418us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:10.919373 (+ 20785us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.146024 (+1226651us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b54000235d0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.146242 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2625408}
I20260620 19:59:12.147737 8691 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:12.148514 8874 log.cc:927] Time spent T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260620 19:59:12.153268 6301 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
W20260620 19:59:12.162832 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.218685 8812 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.219453 6874 leader_election.cc:400] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5.
I20260620 19:59:12.219763 6874 leader_election.cc:403] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:59:12.081453 8776 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.220814 8812 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.221391 8812 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 5
I20260620 19:59:12.221704 8776 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.154199 8871 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:12.222368 8871 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:12.203616 6378 raft_consensus.cc:1240] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260620 19:59:12.058946 6904 rpcz_store.cc:276] 0620 19:59:10.528037 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.690264 (+1162227us) service_pool.cc:224] Handling call
0620 19:59:11.735048 (+ 44784us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:11.745064 (+ 10016us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:11.747194 (+ 2130us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:11.753368 (+ 6174us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:11.796213 (+ 42845us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:11.805633 (+ 9420us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:11.836748 (+ 31115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:12.186306 8728 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:12.213137 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:12.187289 8877 log.cc:927] Time spent T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
W20260620 19:59:12.183815 8873 log.cc:927] Time spent T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260620 19:59:12.219475 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.236315 6230 raft_consensus.cc:1240] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:12.132687 8756 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.224467 6874 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:12.209564 6304 rpcz_store.cc:276] 0620 19:59:11.105985 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.106121 (+ 136us) service_pool.cc:224] Handling call
0620 19:59:11.106391 (+ 270us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:11.127274 (+ 20883us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:11.127304 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:11.127345 (+ 41us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:11.127381 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.141231 (+1013850us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b54001180d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:12.141283 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:12.141419 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47604480}
I20260620 19:59:12.184895 8820 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.234930 6225 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
W20260620 19:59:12.223944 8867 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.405s user 0.001s sys 0.000s
W20260620 19:59:12.276623 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 52215.8us
I20260620 19:59:12.204322 8750 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:12.247519 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.275698 8837 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.210261 6305 rpcz_store.cc:276] 0620 19:59:11.105010 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.107626 (+ 2616us) service_pool.cc:224] Handling call
0620 19:59:11.107978 (+ 352us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:11.108391 (+ 413us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:11.108431 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:11.109012 (+ 581us) log.cc:844] Serialized 26 byte log entry
0620 19:59:11.407006 (+297994us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:11.559960 (+152954us) log.cc:844] Serialized 14 byte log entry
0620 19:59:11.601769 (+ 41809us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:11.601872 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:12.141838 (+539966us) raft_consensus.cc:1685] finished
0620 19:59:12.141861 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:12.141987 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7004,"prepare.queue_time_us":245680,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2244,"raft.queue_time_us":37568,"raft.run_cpu_time_us":450,"raft.run_wall_time_us":68536,"thread_start_us":352822,"threads_started":2,"wal-append.queue_time_us":352639}
I20260620 19:59:12.281076 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.231866 8691 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.161336 8838 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.247944 8829 consensus_meta.cc:287] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.112s user 0.011s sys 0.001s
I20260620 19:59:12.154315 8841 consensus_queue.cc:237] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.293901 6344 connection.cc:256] RPC call timeout handler was delayed by -1.5516361150000648s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2778s for a response
I20260620 19:59:12.209888 8541 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.294473 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.289860 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:12.298454 6344 connection.cc:256] RPC call timeout handler was delayed by -1.4611493659999724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20260620 19:59:12.298775 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:12.294513 6302 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 626) took 2151 ms (client timeout 1900 ms). Trace:
W20260620 19:59:12.299762 6302 rpcz_store.cc:269] 0620 19:59:10.142761 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.163850 (+ 21089us) service_pool.cc:224] Handling call
0620 19:59:12.294297 (+2130447us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54001176d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.294422 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4113043840}
I20260620 19:59:12.207816 8824 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.295159 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1548) took 1190 ms. Trace:
I20260620 19:59:12.303613 6306 rpcz_store.cc:276] 0620 19:59:11.104660 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.107528 (+ 2868us) service_pool.cc:224] Handling call
0620 19:59:11.127476 (+ 19948us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:11.135707 (+ 8231us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:11.135742 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:11.188044 (+ 52302us) log.cc:844] Serialized 26 byte log entry
0620 19:59:11.702235 (+514191us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b4800061a58. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6bd7e41a1 00007fa6bd7db237 00007fa6bd7db0ca 00007fa6bd8d296f 00007fa6bd8d2637 00007fa6bd8e6c5c 00007fa6bd8e6c08 00007fa6bd8e6b88 00007fa6bd8e6b43 00007fa6bd8e5dec 00007fa6bc812f76
0620 19:59:12.195644 (+493409us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:12.195708 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.195804 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:12.294886 (+ 99082us) raft_consensus.cc:1685] finished
0620 19:59:12.294903 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:12.295104 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7360,"raft.queue_time_us":74,"raft.run_cpu_time_us":2615,"raft.run_wall_time_us":562261,"spinlock_wait_cycles":61668608,"thread_start_us":960920,"threads_started":1,"wal-append.queue_time_us":1014656}
I20260620 19:59:12.186627 8740 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.305446 6305 raft_consensus.cc:2368] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5.
W20260620 19:59:12.307390 6344 connection.cc:256] RPC call timeout handler was delayed by -1.440574416000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32970000000000005s for a response
I20260620 19:59:12.236243 8881 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:12.309757 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1770) took 1060 ms. Trace:
I20260620 19:59:12.281262 8474 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.310263 6305 rpcz_store.cc:276] 0620 19:59:11.249438 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.286694 (+1037256us) service_pool.cc:224] Handling call
0620 19:59:12.309682 (+ 22988us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:12.308708 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.285494 8776 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:12.309389 6299 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:12.315172 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.315363 6911 raft_consensus.cc:2468] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:12.223444 8862 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.318327 8753 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.318324 6372 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:12.288916 8787 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.303603 8691 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.352725 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2054) took 1212 ms. Trace:
I20260620 19:59:12.358179 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.314360 8881 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.308908 8821 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:12.349761 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.353421 6909 raft_consensus.cc:1240] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:59:12.366501 8825 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.313292 8738 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.369638 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:12.370103 6910 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:12.354903 6227 rpcz_store.cc:276] 0620 19:59:11.140776 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.168814 (+ 28038us) service_pool.cc:224] Handling call
0620 19:59:11.199419 (+ 30605us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:11.456949 (+257530us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:11.456991 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:11.504379 (+ 47388us) log.cc:844] Serialized 52 byte log entry
0620 19:59:11.746238 (+241859us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:11.809739 (+ 63501us) log.cc:844] Serialized 14 byte log entry
0620 19:59:11.810107 (+ 368us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:11.810200 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:12.352517 (+542317us) raft_consensus.cc:1685] finished
0620 19:59:12.352538 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:12.352670 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":879,"prepare.queue_time_us":63252,"prepare.run_wall_time_us":100986,"raft.queue_time_us":199317,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":117501,"spinlock_wait_cycles":58514176,"thread_start_us":363282,"threads_started":2,"wal-append.queue_time_us":328139}
I20260620 19:59:12.382109 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.382973 6302 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:12.384804 8846 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.319219 8474 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.387744 8474 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.297317 8750 consensus_queue.cc:260] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.303288 8829 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:12.352427 8776 raft_consensus.cc:697] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:12.398200 6910 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:12.354315 8654 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:12.424484 6910 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 669) took 1893 ms (client timeout 1863 ms). Trace:
I20260620 19:59:12.418953 8825 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:12.432772 6910 rpcz_store.cc:269] 0620 19:59:10.530958 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.578326 (+ 47368us) service_pool.cc:224] Handling call
0620 19:59:12.369527 (+1791201us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:59:12.424417 (+ 54890us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5796096}
I20260620 19:59:12.371856 8746 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:12.369015 6906 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 828) took 2316 ms (client timeout 1754 ms). Trace:
W20260620 19:59:12.450428 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 138507us
W20260620 19:59:12.455751 6344 connection.cc:256] RPC call timeout handler was delayed by -1.2424493660000735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
I20260620 19:59:12.456171 6231 raft_consensus.cc:2393] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:12.456244 6344 connection.cc:256] RPC call timeout handler was delayed by -1.1533493660000431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20260620 19:59:12.456748 6344 connection.cc:256] RPC call timeout handler was delayed by -1.1425493660000257s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1876s for a response
W20260620 19:59:12.457165 6344 connection.cc:256] RPC call timeout handler was delayed by -1.1301009239999757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
W20260620 19:59:12.457585 6344 connection.cc:256] RPC call timeout handler was delayed by -1.0522995050000645s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260620 19:59:12.456775 6910 raft_consensus.cc:1240] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260620 19:59:12.458962 6344 leader_election.cc:336] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.522s (SENT)
I20260620 19:59:12.459229 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 666) took 1929 ms. Trace:
I20260620 19:59:12.459576 6910 rpcz_store.cc:276] 0620 19:59:10.530086 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.456226 (+1926140us) service_pool.cc:224] Handling call
0620 19:59:12.456558 (+ 332us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:12.458970 (+ 2412us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.459178 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:12.479754 8838 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.485200 8871 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:12.487681 8824 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:12.489506 8839 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 1.357s user 0.000s sys 0.009s
I20260620 19:59:12.491048 8776 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.492592 8756 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:12.455209 6906 rpcz_store.cc:269] 0620 19:59:10.037296 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.038297 (+ 1001us) service_pool.cc:224] Handling call
0620 19:59:12.313975 (+2275678us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54005302d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.353904 (+ 39929us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":729288576}
I20260620 19:59:12.497978 8829 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.463622 8825 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.503286 6344 leader_election.cc:336] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.765s (SENT)
I20260620 19:59:12.503391 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.446565 8750 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:12.374344 8821 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:12.463061 8728 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.451773 8654 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.512993 6344 leader_election.cc:336] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.864s (SENT)
I20260620 19:59:12.512817 6302 raft_consensus.cc:2376] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5. Re-sending same reply.
W20260620 19:59:12.507560 8737 consensus_meta.cc:287] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 1.507s user 0.008s sys 0.004s
I20260620 19:59:12.503664 8691 leader_election.cc:290] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.454720 8798 consensus_queue.cc:260] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.510478 8809 consensus_meta.cc:287] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 1.457s user 0.019s sys 0.021s
I20260620 19:59:12.517889 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 667) took 1987 ms. Trace:
I20260620 19:59:12.165606 6298 rpcz_store.cc:276] 0620 19:59:10.720412 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.729919 (+ 9507us) service_pool.cc:224] Handling call
0620 19:59:10.772332 (+ 42413us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:10.887468 (+115136us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:10.902937 (+ 15469us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:11.017555 (+114618us) log.cc:844] Serialized 26 byte log entry
0620 19:59:11.193368 (+175813us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:11.193424 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:11.193505 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:12.046954 (+853449us) raft_consensus.cc:1685] finished
0620 19:59:12.054024 (+ 7070us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:12.063799 (+ 9775us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47643,"raft.queue_time_us":175124,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":6043,"thread_start_us":124815,"threads_started":1,"wal-append.queue_time_us":194625}
I20260620 19:59:12.522557 8799 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.524857 8825 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.525185 8781 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:12.525882 8846 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:12.526996 8711 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.527259 8837 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.527817 8583 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.528198 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 2060) took 1358 ms. Trace:
I20260620 19:59:12.529433 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:12.530633 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 827) took 2493 ms. Trace:
I20260620 19:59:12.531905 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:12.532305 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.533478 8541 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.533874 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.534328 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:12.534981 8802 consensus_meta.cc:287] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 1.946s user 0.016s sys 0.012s
I20260620 19:59:12.535283 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 825) took 2500 ms. Trace:
W20260620 19:59:12.537366 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.537999 6344 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:12.538754 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.504454 6902 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:12.558687 6877 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:12.536124 8760 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.507769 8833 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:12.560596 6911 rpcz_store.cc:276] 0620 19:59:10.530368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.481186 (+950818us) service_pool.cc:224] Handling call
0620 19:59:12.517695 (+1036509us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x7b5400531454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.517809 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1645216512}
I20260620 19:59:12.583122 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:12.671835 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.671531 8802 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.692984 8891 log.cc:927] Time spent T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
W20260620 19:59:12.675462 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.540333 8735 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.695268 6379 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:12.692557 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.702531 6302 raft_consensus.cc:2393] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:12.695535 6307 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:59:12.671536 6299 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:59:12.684820 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 126262us
I20260620 19:59:12.697824 6380 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:12.697696 6225 rpcz_store.cc:276] 0620 19:59:11.169557 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.208176 (+ 38619us) service_pool.cc:224] Handling call
0620 19:59:12.528036 (+1319860us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b5400077954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.528141 (+ 105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2044866816}
I20260620 19:59:12.697356 6375 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:12.704028 6877 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:59:12.639035 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:12.728160 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:12.733321 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.733515 8583 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.711445 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.735149 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.559882 8824 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.557304 8728 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.694422 8738 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:12.559744 8834 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.559779 8474 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.734695 8795 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260620 19:59:12.738240 8889 log.cc:927] Time spent T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20260620 19:59:12.694034 8881 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.739228 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.775663 6307 raft_consensus.cc:2410] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260620 19:59:12.705332 6877 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:12.560181 8737 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.694484 8862 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.556495 8756 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:12.744995 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.745061 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 659) took 2216 ms. Trace:
I20260620 19:59:12.783905 6909 rpcz_store.cc:276] 0620 19:59:10.528263 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.731600 (+1203337us) service_pool.cc:224] Handling call
0620 19:59:12.318797 (+587197us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:12.362677 (+ 43880us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.744783 (+382106us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b54005302d0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.744984 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1329369856}
I20260620 19:59:12.671180 6905 rpcz_store.cc:276] 0620 19:59:10.035050 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.035856 (+ 806us) service_pool.cc:224] Handling call
0620 19:59:10.613813 (+577957us) spinlock_profiling.cc:243] Waited 8.44 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b579169c 00007fa6b579125b 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:10.634769 (+ 20956us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:10.762988 (+128219us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:10.765562 (+ 2574us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:10.897431 (+131869us) log.cc:844] Serialized 26 byte log entry
0620 19:59:11.098811 (+201380us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:11.114466 (+ 15655us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.366292 (+1251826us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b18003b9aa0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:12.534944 (+168652us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54005ee554. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:12.535002 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:12.535054 (+ 52us) raft_consensus.cc:1685] finished
0620 19:59:12.535073 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:12.535223 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4984,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":9310,"spinlock_wait_cycles":194745344,"thread_start_us":74473,"threads_started":1,"wal-append.queue_time_us":161188}
I20260620 19:59:12.696832 8691 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.673910 8809 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:12.696843 8606 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:12.661861 8798 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:12.751513 6379 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:12.756496 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.767437 8654 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.671200 6903 rpcz_store.cc:276] 0620 19:59:10.036927 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.039163 (+ 2236us) service_pool.cc:224] Handling call
0620 19:59:10.293549 (+254386us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:10.293775 (+ 226us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:10.568267 (+274492us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:10.570875 (+ 2608us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:10.719922 (+149047us) log.cc:844] Serialized 52 byte log entry
0620 19:59:10.950215 (+230293us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:10.954529 (+ 4314us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:10.974530 (+ 20001us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:12.478046 (+1503516us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:12.530364 (+ 52318us) raft_consensus.cc:1685] finished
0620 19:59:12.530391 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:12.530534 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":87781,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":8627,"spinlock_wait_cycles":14582272,"thread_start_us":352433,"threads_started":2,"wal-append.queue_time_us":306990}
I20260620 19:59:12.768946 8541 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:12.776669 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 71417.3us
I20260620 19:59:12.740399 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.750876 8824 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.695614 8781 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.781092 6380 raft_consensus.cc:2468] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:12.586316 8846 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.782141 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.806751 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.807075 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 2066) took 1300 ms. Trace:
I20260620 19:59:12.807354 8846 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.808698 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:12.782325 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 269279us
W20260620 19:59:12.809662 6344 leader_election.cc:336] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.876s (SENT)
I20260620 19:59:12.811185 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2057) took 1664 ms. Trace:
I20260620 19:59:12.811671 6224 rpcz_store.cc:276] 0620 19:59:11.146727 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.176273 (+ 29546us) service_pool.cc:224] Handling call
0620 19:59:11.209717 (+ 33444us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:12.216190 (+1006473us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.810974 (+594784us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54001d51d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:12.811129 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2308665088}
I20260620 19:59:12.762329 8795 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:12.813072 8795 raft_consensus.cc:3060] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:12.814785 8842 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.816229 8728 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.789485 8809 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.819013 8809 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.819249 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1935) took 1409 ms. Trace:
I20260620 19:59:12.819607 6378 rpcz_store.cc:276] 0620 19:59:11.409415 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.409867 (+ 452us) service_pool.cc:224] Handling call
0620 19:59:11.456531 (+ 46664us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:12.226841 (+770310us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.819018 (+592177us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54000456d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:12.819195 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1716984576}
I20260620 19:59:12.761875 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.788023 6302 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:12.782869 8738 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:12.700397 8833 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.791374 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.793834 6877 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:12.746172 6904 raft_consensus.cc:1240] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260620 19:59:12.798751 8821 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.799830 8583 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.800688 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.837634 6299 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:12.773098 8474 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.839504 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:12.807530 6307 raft_consensus.cc:2410] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260620 19:59:12.696677 8837 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.840667 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 46939us
I20260620 19:59:12.841871 6910 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:12.809749 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.807500 6231 rpcz_store.cc:276] 0620 19:59:11.545988 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.723350 (+177362us) service_pool.cc:224] Handling call
0620 19:59:12.806910 (+1083560us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b54000fb9d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.807023 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1659974528}
I20260620 19:59:12.584259 8711 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.844228 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.845791 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.847332 6910 raft_consensus.cc:1240] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:12.845872 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:12.827394 6344 leader_election.cc:336] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.778s (SENT)
W20260620 19:59:12.827984 6907 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 664) took 2298 ms (client timeout 1947 ms). Trace:
I20260620 19:59:12.787997 8691 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:12.785579 8862 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:12.791155 8606 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.834781 6305 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:12.835721 8776 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:12.852221 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:12.836388 8833 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.764469 8834 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.786703 8756 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:12.846078 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.855669 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.856402 6299 raft_consensus.cc:2393] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:12.856436 6273 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:12.857998 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.858070 6380 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:12.858186 6273 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:12.840428 8541 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.848095 6304 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:12.848151 6344 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:12.843971 8740 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:12.849417 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:12.864674 6910 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:12.865298 6273 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:12.865609 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:12.866245 6910 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:12.866662 6273 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:12.866994 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:12.850512 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.852661 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.853965 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:12.868371 6305 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:12.868423 6374 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:12.854102 6875 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:12.840471 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:12.860621 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:12.848227 6907 rpcz_store.cc:269] 0620 19:59:10.529525 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.700732 (+171207us) service_pool.cc:224] Handling call
0620 19:59:12.827817 (+2127085us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400587854. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:12.827921 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1170737920}
W20260620 19:59:12.822511 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.849587 8871 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.972560 6910 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:12.940088 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:12.952737 6305 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
W20260620 19:59:12.918958 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 218336us
W20260620 19:59:12.976348 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 122300us
W20260620 19:59:12.897274 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:12.934799 8898 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:12.978868 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:12.936112 8895 log.cc:927] Time spent T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260620 19:59:12.979650 8711 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.935494 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:12.980103 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:12.920125 8751 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:12.980821 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:12.982380 8737 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:12.983247 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 665) took 2453 ms. Trace:
I20260620 19:59:12.983661 6904 rpcz_store.cc:276] 0620 19:59:10.529801 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.276826 (+1747025us) service_pool.cc:224] Handling call
0620 19:59:12.308031 (+ 31205us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:12.842695 (+534664us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:12.983009 (+140314us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b54006e0354. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:12.983189 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":982888832}
W20260620 19:59:12.977802 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 150428us
I20260620 19:59:12.983850 8795 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.909399 6273 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:59:12.967234 8896 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260620 19:59:12.857174 8802 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:12.921296 8750 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260620 19:59:13.000209 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 90913us
I20260620 19:59:12.920296 8753 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:12.921181 8708 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260620 19:59:13.000777 6344 leader_election.cc:336] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.218s (SENT)
I20260620 19:59:13.001976 6273 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:12.857261 8776 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.871594 8542 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:12.921392 8814 raft_consensus.cc:2804] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:12.922652 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:12.983772 8474 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.936582 8824 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:12.967686 8691 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.001200 8750 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:12.936403 8834 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.002862 6906 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:13.087812 8750 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:13.088187 8756 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.088785 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:13.089731 6906 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:12.876610 8541 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:12.863191 8728 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:13.090955 8728 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:13.066236 8760 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:13.069188 8814 raft_consensus.cc:697] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:13.093006 8814 consensus_queue.cc:237] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:13.094830 6905 raft_consensus.cc:2393] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:13.095314 8541 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.095846 8738 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:13.096256 6905 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260620 19:59:13.096733 6905 pending_rounds.cc:85] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting all ops after (but not including) 5
I20260620 19:59:13.097013 6905 pending_rounds.cc:107] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260620 19:59:13.097312 6905 raft_consensus.cc:2889] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:13.097640 8750 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:12.989756 8751 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:12.988708 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:13.020985 6907 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:13.023164 8809 raft_consensus.cc:2804] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:13.025522 8836 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:13.011557 8708 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:13.027674 8783 consensus_queue.cc:1059] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:13.003248 8779 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:13.099143 8862 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.122625 8708 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:13.099458 8760 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:13.127177 6298 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:13.122630 8836 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:13.127966 8760 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.128204 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:13.108734 8812 consensus_meta.cc:287] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.887s user 0.011s sys 0.000s
I20260620 19:59:13.129068 8812 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5.
I20260620 19:59:13.129392 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:13.099823 6302 raft_consensus.cc:2410] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260620 19:59:13.131138 8751 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:13.132012 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1939) took 1663 ms. Trace:
I20260620 19:59:13.132210 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:13.132368 6377 rpcz_store.cc:276] 0620 19:59:11.468797 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.468965 (+ 168us) service_pool.cc:224] Handling call
0620 19:59:11.486920 (+ 17955us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:11.797756 (+310836us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:13.131793 (+1334037us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b5400064d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:13.131954 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":713395456}
I20260620 19:59:13.133252 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:13.099212 8728 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.134229 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:13.135598 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2063) took 1817 ms. Trace:
I20260620 19:59:13.136548 8750 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.122080 8837 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.143961 6230 rpcz_store.cc:276] 0620 19:59:11.318138 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.352719 (+ 34581us) service_pool.cc:224] Handling call
0620 19:59:11.353029 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:12.237100 (+884071us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:13.135390 (+898290us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b540004c4d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:13.135543 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2028643584}
I20260620 19:59:13.150055 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:13.150643 6196 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:13.155352 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 142386us
I20260620 19:59:13.168602 6196 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:13.170081 6196 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:13.172937 8708 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:13.174817 8862 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:13.175351 8862 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:13.175796 8862 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:13.178773 8842 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:13.179204 8842 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:13.179543 8829 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:13.179539 8842 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:13.078565 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:13.187896 6196 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:13.146061 8795 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:13.189832 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 189036us
W20260620 19:59:13.192909 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.161885 8756 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:13.138562 8542 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:13.227969 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:13.228921 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.193476 6904 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:13.200376 8827 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:13.200691 8691 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:13.253698 6902 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 656) took 2726 ms (client timeout 1600 ms). Trace:
I20260620 19:59:13.254343 8542 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.254861 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1551) took 2149 ms. Trace:
I20260620 19:59:13.258876 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:13.230396 8781 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.201133 8831 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260620 19:59:13.260155 8728 raft_consensus.cc:3055] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:13.260097 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:13.251230 6905 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:13.261188 8751 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:13.262121 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.267380 6906 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:13.190580 8882 consensus_queue.cc:1059] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
W20260620 19:59:13.269701 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.296269 8882 raft_consensus.cc:3055] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:13.296636 8882 raft_consensus.cc:740] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:13.297441 8882 consensus_queue.cc:260] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:13.240738 6344 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:59:13.299185 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 58519.1us
I20260620 19:59:13.281332 6304 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:13.280246 6303 rpcz_store.cc:276] 0620 19:59:11.105672 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.105894 (+ 222us) service_pool.cc:224] Handling call
0620 19:59:11.120702 (+ 14808us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:11.387131 (+266429us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:13.254629 (+1867498us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b5400322dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:13.254798 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":351169408}
I20260620 19:59:13.227471 8779 raft_consensus.cc:697] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:13.273597 8900 log.cc:927] Time spent T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260620 19:59:13.302459 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 636) took 2190 ms. Trace:
I20260620 19:59:13.302892 6304 rpcz_store.cc:276] 0620 19:59:11.112513 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.980653 (+1868140us) service_pool.cc:224] Handling call
0620 19:59:13.258711 (+278058us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b5800003098. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:59:13.302390 (+ 43679us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17950080}
I20260620 19:59:13.260478 8708 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.280308 8809 raft_consensus.cc:697] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:13.294966 6904 raft_consensus.cc:2393] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:13.299610 6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1768) took 2090 ms. Trace:
I20260620 19:59:13.285254 8728 raft_consensus.cc:740] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:13.212996 8756 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:13.300302 6344 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:13.304702 6273 rpcz_store.cc:276] 0620 19:59:11.209171 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:13.299562 (+2090391us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260620 19:59:13.308019 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:13.308578 6303 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:13.315429 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:13.316241 6230 raft_consensus.cc:2410] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260620 19:59:13.317044 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:13.280956 8827 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:13.317591 6231 raft_consensus.cc:2376] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 9. Re-sending same reply.
I20260620 19:59:13.319072 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:13.319598 6230 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:13.299863 8882 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:13.318316 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:13.325476 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:13.325932 6231 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 9. Current term is 10. Ops: []
W20260620 19:59:13.327818 6273 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:52348 dropped due to backpressure: service queue is full with 50 items
W20260620 19:59:13.278687 6902 rpcz_store.cc:269] 0620 19:59:10.526795 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.533058 (+ 6263us) service_pool.cc:224] Handling call
0620 19:59:13.253516 (+2720458us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540052c454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:13.253626 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3476447872}
I20260620 19:59:13.323092 6303 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:13.324358 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:13.330744 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:13.328539 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:13.327850 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:13.332188 6229 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:13.332403 6231 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:13.304252 8781 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260620 19:59:13.322978 6345 connection.cc:256] RPC call timeout handler was delayed by -1.8203021769999168s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
W20260620 19:59:13.333516 6345 connection.cc:256] RPC call timeout handler was delayed by -1.1747593499999311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
W20260620 19:59:13.333988 6345 connection.cc:256] RPC call timeout handler was delayed by -1.0136593500000117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
I20260620 19:59:13.334163 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:13.334492 6345 connection.cc:256] RPC call timeout handler was delayed by -1.0136593500000117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
I20260620 19:59:13.313905 8897 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:13.333909 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:13.335657 6232 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:13.337402 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:13.338682 6224 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:13.340684 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:13.338060 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:13.334493 6298 raft_consensus.cc:2368] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 6.
I20260620 19:59:13.324980 8776 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:13.326774 8606 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.336673 8837 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:13.335024 8781 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:13.348501 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.349581 6377 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:13.348163 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:13.344611 8798 consensus_meta.cc:287] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.553s user 0.008s sys 0.002s
I20260620 19:59:13.351364 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:13.354130 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.354313 6372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1586) took 3048 ms. Trace:
W20260620 19:59:13.355795 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.356712 6378 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:13.346216 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:13.376725 6908 raft_consensus.cc:2410] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260620 19:59:13.378923 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:13.379415 6377 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:13.357508 6372 rpcz_store.cc:276] 0620 19:59:10.305719 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.359967 (+ 54248us) service_pool.cc:224] Handling call
0620 19:59:10.388557 (+ 28590us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:12.350334 (+1961777us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:13.354046 (+1003712us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400044cd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:13.354232 (+ 186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4416410368}
I20260620 19:59:13.381690 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.350694 8781 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:13.382485 6379 raft_consensus.cc:2410] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:13.358515 8728 consensus_queue.cc:260] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:13.390964 6905 raft_consensus.cc:2393] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:13.391942 8827 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.392073 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:13.392735 6905 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:13.393647 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:13.390280 6231 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:13.395936 6378 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:13.328218 6273 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:52348 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 650)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1572)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1565)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1827)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1563)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1793)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 648)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 646)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1832)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:60942 (request call id 23)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1769)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1772)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 637)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 638)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1776)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1777)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1779)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1780)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1558)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1787)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1788)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1789)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1799)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1808)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1820)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1564)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1566)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1567)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1571)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1829)
I20260620 19:59:13.382829 8643 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:13.346856 8779 consensus_queue.cc:237] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:13.402927 8643 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:13.404137 8728 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:13.400733 6273 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:52350 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 650)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1572)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1565)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1827)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1563)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1793)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 648)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 646)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1832)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:60942 (request call id 23)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1769)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1772)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 637)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 638)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1776)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1777)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1779)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1780)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1558)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1787)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1788)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1789)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1799)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1808)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1820)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1564)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1566)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1567)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1571)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1836)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1574)
I20260620 19:59:13.412246 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:13.414775 6195 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:13.354190 8837 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:13.416898 6224 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:13.416479 8708 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:13.419031 6372 raft_consensus.cc:1240] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260620 19:59:13.420291 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:13.420814 6372 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:13.423211 8756 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.406533 8862 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:13.426796 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.418512 8708 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:13.430974 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.434337 6381 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
W20260620 19:59:13.436383 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.437188 6377 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:13.437328 6379 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260620 19:59:13.437179 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.438691 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.439160 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:13.439481 6376 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:13.440068 6378 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:13.442952 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:13.443514 6377 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 6
I20260620 19:59:13.446730 6375 raft_consensus.cc:1240] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:13.357097 8809 consensus_queue.cc:237] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:13.450207 6300 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1554) took 2343 ms (client timeout 2214 ms). Trace:
W20260620 19:59:13.450578 6300 rpcz_store.cc:269] 0620 19:59:11.110361 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:11.110530 (+ 169us) service_pool.cc:224] Handling call
0620 19:59:13.450024 (+2339494us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54001f6654. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:13.450139 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1358347008}
I20260620 19:59:13.453907 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:13.454491 6902 raft_consensus.cc:2376] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7. Re-sending same reply.
I20260620 19:59:13.455497 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:13.416311 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:13.455756 8837 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.456327 6905 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:13.462123 6377 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:13.440557 6273 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:52350 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 650)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1572)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1565)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1827)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1563)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1793)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 648)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1835)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 646)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1832)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 651)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:60942 (request call id 23)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 633)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1769)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1772)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 637)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 638)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1776)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1777)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1779)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1780)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1558)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1787)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1788)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1789)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1799)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1808)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1820)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1564)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1566)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1567)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1571)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1829)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1836)
I20260620 19:59:13.417476 8751 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:13.417189 8827 raft_consensus.cc:2764] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20260620 19:59:13.486195 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.486908 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.435863 8618 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:13.489944 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:13.427335 8730 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260620 19:59:13.490659 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.047302 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:13.784816 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.048040 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:13.836659 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:14.048534 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:14.049197 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:13.512144 8893 log.cc:927] Time spent T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.728s user 0.010s sys 0.008s
I20260620 19:59:13.455756 8708 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:13.482713 8691 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 19:59:14.050894 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.507373 8842 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.051798 8618 raft_consensus.cc:697] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:14.053161 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:14.053632 8842 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:14.053993 6381 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:13.504386 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:13.820310 8815 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:14.054909 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:14.055423 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.055933 6381 raft_consensus.cc:2393] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:14.064661 6373 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260620 19:59:14.065315 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:14.066524 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:14.066804 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:13.509128 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:13.606424 6372 raft_consensus.cc:2468] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:13.779234 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:13.752053 8779 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.063403 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.063761 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.050889 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.080893 6231 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 6
I20260620 19:59:14.052901 8618 consensus_queue.cc:237] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:14.047955 6226 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:13.650142 8751 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:13.685526 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.085347 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:13.490733 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.087598 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.087906 8779 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:14.088482 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:13.549397 8862 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:14.088740 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.089377 6298 raft_consensus.cc:2410] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260620 19:59:14.089555 6902 raft_consensus.cc:2393] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 4: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:14.090046 8779 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.074638 6381 raft_consensus.cc:3055] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:14.090735 6381 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:14.090898 6905 raft_consensus.cc:3060] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:14.091502 6381 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:14.091943 6908 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 662) took 3563 ms (client timeout 1845 ms). Trace:
W20260620 19:59:13.921689 8908 log.cc:927] Time spent T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20260620 19:59:14.056156 6224 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:14.092317 6908 rpcz_store.cc:269] 0620 19:59:10.528937 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:10.575085 (+ 46148us) service_pool.cc:224] Handling call
0620 19:59:10.760362 (+185277us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:59:14.091752 (+3331390us) spinlock_profiling.cc:243] Waited 2.59 s on lock 0x7b540068e2d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.091871 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5959497856}
I20260620 19:59:14.093570 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:13.977715 6903 consensus_meta.cc:287] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.825s user 0.009s sys 0.001s
I20260620 19:59:13.521776 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.097326 6225 raft_consensus.cc:2410] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:14.080997 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 677) took 1237 ms. Trace:
I20260620 19:59:14.081372 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:13.656756 6378 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:14.082681 6228 raft_consensus.cc:2393] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:13.755597 6345 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:14.087150 8751 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:13.997289 8781 consensus_meta.cc:287] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.600s user 0.008s sys 0.001s
I20260620 19:59:14.094106 6908 raft_consensus.cc:2376] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5. Re-sending same reply.
I20260620 19:59:14.088754 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.101722 6226 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:14.102087 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.105142 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.105801 6908 raft_consensus.cc:2376] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6. Re-sending same reply.
I20260620 19:59:14.106773 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.092411 6381 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:14.095223 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.074692 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1595) took 1235 ms. Trace:
I20260620 19:59:14.110718 6380 rpcz_store.cc:276] 0620 19:59:12.839760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.839882 (+ 122us) service_pool.cc:224] Handling call
0620 19:59:14.064189 (+1224307us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b54001220d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.074633 (+ 10444us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12232064}
I20260620 19:59:14.098143 6299 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 2.
I20260620 19:59:14.112067 6224 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:14.112157 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.112473 6231 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:14.112926 6299 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 3: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:14.112967 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.098484 6907 rpcz_store.cc:276] 0620 19:59:12.843256 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.986626 (+143370us) service_pool.cc:224] Handling call
0620 19:59:12.991366 (+ 4740us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:13.122296 (+130930us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.080775 (+958479us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b54004b8054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:14.080943 (+ 168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51163904}
I20260620 19:59:14.113323 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.113605 6224 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:14.099671 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.114410 6305 raft_consensus.cc:2393] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:14.115957 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.116474 6305 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:14.090319 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.055100 8815 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:14.090451 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.117919 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.127092 6302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1811) took 1317 ms. Trace:
I20260620 19:59:14.127468 6302 rpcz_store.cc:276] 0620 19:59:12.809878 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.858956 (+ 49078us) service_pool.cc:224] Handling call
0620 19:59:14.126917 (+1267961us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54001167d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.127034 (+ 117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281023744}
I20260620 19:59:14.119261 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.129041 6299 raft_consensus.cc:2393] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:14.129762 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.129823 6224 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 3.
I20260620 19:59:14.130519 6299 raft_consensus.cc:2410] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:14.120221 6195 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:14.130843 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.121665 8751 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:14.099200 6903 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:14.121970 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.104703 8781 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.123437 8787 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:14.123624 6381 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:14.123770 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 673) took 1285 ms. Trace:
I20260620 19:59:14.137584 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.137449 6911 rpcz_store.cc:276] 0620 19:59:12.837941 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.838050 (+ 109us) service_pool.cc:224] Handling call
0620 19:59:12.838317 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:13.148953 (+310636us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:13.149001 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:13.149064 (+ 63us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:13.149110 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.123554 (+974444us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54006f4ad4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:14.123595 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:14.123718 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":713669376}
I20260620 19:59:14.124607 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.139341 6300 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:14.118470 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.131415 6304 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:14.131516 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.140789 6908 raft_consensus.cc:2410] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260620 19:59:14.141258 6907 raft_consensus.cc:2393] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:14.132050 6195 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:14.133255 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.141304 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.142264 6299 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:14.133828 8691 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:14.125957 6305 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:14.136775 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.148016 6302 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:14.148039 6374 raft_consensus.cc:1240] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:14.134522 6231 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:14.147351 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.148572 6195 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:14.149432 6300 raft_consensus.cc:2410] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260620 19:59:14.149333 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.150413 6231 raft_consensus.cc:2410] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260620 19:59:14.151060 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.151556 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 841) took 1012 ms. Trace:
I20260620 19:59:14.152012 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.152060 6908 rpcz_store.cc:276] 0620 19:59:13.139302 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.131305 (+992003us) service_pool.cc:224] Handling call
0620 19:59:14.151427 (+ 20122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.152588 6299 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:14.153343 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1791) took 1792 ms. Trace:
I20260620 19:59:14.153383 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.153604 6299 rpcz_store.cc:276] 0620 19:59:12.361085 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.151810 (+1790725us) service_pool.cc:224] Handling call
0620 19:59:14.153304 (+ 1494us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.154397 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 846) took 1012 ms. Trace:
I20260620 19:59:14.155159 6907 rpcz_store.cc:276] 0620 19:59:13.141892 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.113840 (+971948us) service_pool.cc:224] Handling call
0620 19:59:14.154321 (+ 40481us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.155393 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.157582 6299 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:14.132797 8862 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260620 19:59:14.158107 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1806) took 1374 ms. Trace:
I20260620 19:59:14.147107 6298 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:14.161799 6300 rpcz_store.cc:276] 0620 19:59:12.783513 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.147008 (+1363495us) service_pool.cc:224] Handling call
0620 19:59:14.158053 (+ 11045us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.157332 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.162700 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.163110 6911 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:14.157337 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 401753us
I20260620 19:59:14.159786 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.132951 8864 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260620 19:59:14.159179 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.163995 6908 raft_consensus.cc:2376] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7. Re-sending same reply.
I20260620 19:59:14.161252 6195 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:14.160315 6902 raft_consensus.cc:2410] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260620 19:59:14.164907 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.166312 6231 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:14.168548 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.169253 6300 raft_consensus.cc:2393] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:14.171286 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.171974 6907 raft_consensus.cc:2393] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:14.172312 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 848) took 1018 ms. Trace:
I20260620 19:59:14.172587 6907 rpcz_store.cc:276] 0620 19:59:13.153534 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.156826 (+1003292us) service_pool.cc:224] Handling call
0620 19:59:14.172263 (+ 15437us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.156029 8827 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:14.156239 8808 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:14.177538 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 683) took 1332 ms. Trace:
I20260620 19:59:14.178074 6910 rpcz_store.cc:276] 0620 19:59:12.845191 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.866795 (+ 21604us) service_pool.cc:224] Handling call
0620 19:59:14.177341 (+1310546us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54005302d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.177464 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241254784}
I20260620 19:59:14.175009 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1793) took 1700 ms. Trace:
I20260620 19:59:14.179826 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.180238 6300 rpcz_store.cc:276] 0620 19:59:12.474695 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.168348 (+1693653us) service_pool.cc:224] Handling call
0620 19:59:14.174951 (+ 6603us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.180778 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 855) took 1006 ms. Trace:
I20260620 19:59:14.181114 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.181339 6908 rpcz_store.cc:276] 0620 19:59:13.170322 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.164707 (+994385us) service_pool.cc:224] Handling call
0620 19:59:14.177092 (+ 12385us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.181835 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1562) took 1017 ms. Trace:
I20260620 19:59:14.182297 6300 rpcz_store.cc:276] 0620 19:59:13.164070 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.180907 (+1016837us) service_pool.cc:224] Handling call
0620 19:59:14.181785 (+ 878us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.176614 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.183234 6908 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:14.188005 8912 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:14.188467 8912 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:14.188827 8912 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:14.196053 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 689) took 1199 ms. Trace:
I20260620 19:59:14.196568 6908 rpcz_store.cc:276] 0620 19:59:12.988117 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.182410 (+1194293us) service_pool.cc:224] Handling call
0620 19:59:14.182845 (+ 435us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.187103 (+ 4258us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.187344 (+ 241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.202905 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.203576 6907 raft_consensus.cc:2393] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:14.205367 6299 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:14.205883 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1809) took 1396 ms. Trace:
I20260620 19:59:14.170799 8691 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:14.206169 6299 rpcz_store.cc:276] 0620 19:59:12.809347 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.155112 (+1345765us) service_pool.cc:224] Handling call
0620 19:59:14.205830 (+ 50718us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.206187 6231 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:14.209125 6911 raft_consensus.cc:2410] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260620 19:59:14.209513 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 857) took 1016 ms. Trace:
I20260620 19:59:14.209754 6911 rpcz_store.cc:276] 0620 19:59:13.193013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.142865 (+949852us) service_pool.cc:224] Handling call
0620 19:59:14.209470 (+ 66605us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.183248 6304 raft_consensus.cc:2410] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260620 19:59:14.203137 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 845) took 1061 ms. Trace:
I20260620 19:59:14.211210 6910 rpcz_store.cc:276] 0620 19:59:13.141360 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.179530 (+1038170us) service_pool.cc:224] Handling call
0620 19:59:14.203077 (+ 23547us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.183668 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.212632 6300 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:14.213869 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.214844 6225 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:14.214800 6299 raft_consensus.cc:2376] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7. Re-sending same reply.
I20260620 19:59:14.215512 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1821) took 1216 ms. Trace:
I20260620 19:59:14.215672 6911 raft_consensus.cc:1240] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:14.215930 6299 rpcz_store.cc:276] 0620 19:59:12.999094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.213658 (+1214564us) service_pool.cc:224] Handling call
0620 19:59:14.215446 (+ 1788us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.216259 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 850) took 1048 ms. Trace:
I20260620 19:59:14.216872 6907 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:14.216903 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.217612 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 844) took 1076 ms. Trace:
I20260620 19:59:14.217957 6907 rpcz_store.cc:276] 0620 19:59:13.141064 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.204027 (+1062963us) service_pool.cc:224] Handling call
0620 19:59:14.216707 (+ 12680us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.217403 (+ 696us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.217559 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":830336}
I20260620 19:59:14.194514 8808 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:14.219076 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.219514 6299 raft_consensus.cc:2393] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:14.216619 6911 rpcz_store.cc:276] 0620 19:59:13.168397 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.210211 (+1041814us) service_pool.cc:224] Handling call
0620 19:59:14.210528 (+ 317us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.216047 (+ 5519us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.216214 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.218921 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.222117 6223 raft_consensus.cc:2393] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:14.222257 6907 raft_consensus.cc:1240] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:14.210804 8691 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:14.223923 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.227108 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 854) took 1057 ms. Trace:
I20260620 19:59:14.228693 6299 raft_consensus.cc:2410] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:14.228427 6907 rpcz_store.cc:276] 0620 19:59:13.170009 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.219544 (+1049535us) service_pool.cc:224] Handling call
0620 19:59:14.222098 (+ 2554us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.222631 (+ 533us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.227046 (+ 4415us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.232483 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.233032 6304 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:14.233448 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 646) took 1077 ms. Trace:
I20260620 19:59:14.233714 6304 rpcz_store.cc:276] 0620 19:59:13.156480 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.218785 (+1062305us) service_pool.cc:224] Handling call
0620 19:59:14.233399 (+ 14614us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.216477 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.206278 6905 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:14.242707 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.237169 6911 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:14.220541 8808 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:14.243649 6299 raft_consensus.cc:1240] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:14.243356 6225 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:14.244695 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 632) took 3133 ms. Trace:
I20260620 19:59:14.245023 6299 rpcz_store.cc:276] 0620 19:59:11.111091 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.241682 (+3130591us) service_pool.cc:224] Handling call
0620 19:59:14.242379 (+ 697us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.244472 (+ 2093us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.244645 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.246038 6907 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:14.246588 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 691) took 1051 ms. Trace:
I20260620 19:59:14.247393 8815 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.244131 6902 raft_consensus.cc:2376] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4. Re-sending same reply.
I20260620 19:59:14.249341 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 856) took 1061 ms. Trace:
I20260620 19:59:14.254124 6300 raft_consensus.cc:2410] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:14.231928 8691 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:14.255136 6907 rpcz_store.cc:276] 0620 19:59:13.194976 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.229301 (+1034325us) service_pool.cc:224] Handling call
0620 19:59:14.241534 (+ 12233us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.246398 (+ 4864us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.246549 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.256726 6911 rpcz_store.cc:276] 0620 19:59:13.187510 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.221392 (+1033882us) service_pool.cc:224] Handling call
0620 19:59:14.236997 (+ 15605us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.243366 (+ 6369us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.249286 (+ 5920us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.257477 6196 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:14.258590 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 672) took 1421 ms. Trace:
I20260620 19:59:14.258993 6909 rpcz_store.cc:276] 0620 19:59:12.837674 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.838016 (+ 342us) service_pool.cc:224] Handling call
0620 19:59:12.846253 (+ 8237us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:12.846628 (+ 375us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:12.846654 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:12.847169 (+ 515us) log.cc:844] Serialized 52 byte log entry
0620 19:59:12.934686 (+ 87517us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:13.190401 (+255715us) log.cc:844] Serialized 14 byte log entry
0620 19:59:13.199708 (+ 9307us) log.cc:844] Serialized 14 byte log entry
0620 19:59:14.245275 (+1045567us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b1800248f20. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b574340a 00007fa6bee885d9 00007fa6b57cb240 00007fa6b57c5ce4 00007fa6b57c5c55 00007fa6b88b2440 00007fa6b88c4af6 00007fa6b88d5667 00007fa6b88d55e8
0620 19:59:14.245438 (+ 163us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.245490 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:14.258397 (+ 12907us) raft_consensus.cc:1685] finished
0620 19:59:14.258416 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:14.258540 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1038482,"prepare.queue_time_us":328350,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":60,"raft.queue_time_us":1065841,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":11753,"spinlock_wait_cycles":36038528,"thread_start_us":286386,"threads_started":2,"wal-append.queue_time_us":139075}
I20260620 19:59:14.260780 6304 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:59:14.236238 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.261376 6304 raft_consensus.cc:606] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:14.264030 6304 raft_consensus.cc:614] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:14.261672 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 648) took 1104 ms. Trace:
W20260620 19:59:14.264360 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:14.264585 6300 rpcz_store.cc:276] 0620 19:59:13.157107 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.183465 (+1026358us) service_pool.cc:224] Handling call
0620 19:59:14.260676 (+ 77211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.266080 6381 raft_consensus.cc:1275] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:59:14.266348 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:14.266921 6378 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:14.267096 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.268682 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1795) took 1752 ms. Trace:
I20260620 19:59:14.269032 6306 rpcz_store.cc:276] 0620 19:59:12.516851 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:13.149745 (+632894us) service_pool.cc:224] Handling call
0620 19:59:14.268509 (+1118764us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54001176d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.268619 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2172660096}
I20260620 19:59:14.270126 6306 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 5. Ops: []
I20260620 19:59:14.270748 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1769) took 3045 ms. Trace:
I20260620 19:59:14.271034 6306 rpcz_store.cc:276] 0620 19:59:11.225105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.269743 (+3044638us) service_pool.cc:224] Handling call
0620 19:59:14.269983 (+ 240us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.270442 (+ 459us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.270582 (+ 140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.272047 6306 raft_consensus.cc:1240] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:14.272513 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1772) took 2994 ms. Trace:
I20260620 19:59:14.272806 6306 rpcz_store.cc:276] 0620 19:59:11.277804 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.271678 (+2993874us) service_pool.cc:224] Handling call
0620 19:59:14.271907 (+ 229us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.272338 (+ 431us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.272473 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.274477 8864 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:14.274909 8864 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:14.275108 8901 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.275324 8864 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:14.275884 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.276472 6224 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:14.277437 6380 raft_consensus.cc:1275] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260620 19:59:14.275641 8901 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.264285 6196 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:14.281437 6273 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:14.277411 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.282546 6273 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:14.279915 6230 raft_consensus.cc:2393] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 10: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:14.283315 6273 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:14.284248 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.284933 6230 raft_consensus.cc:2393] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:14.285346 6299 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:14.262876 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 698) took 1029 ms. Trace:
I20260620 19:59:14.280644 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.285938 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 633) took 3174 ms. Trace:
I20260620 19:59:14.283351 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.297479 6299 rpcz_store.cc:276] 0620 19:59:11.111618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.245785 (+3134167us) service_pool.cc:224] Handling call
0620 19:59:14.285201 (+ 39416us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.285729 (+ 528us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.285888 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.298659 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:14.286453 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.286772 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.285593 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.284933 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:60942 (request call id 23) took 1788 ms. Trace:
I20260620 19:59:14.292450 6196 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:14.303660 6196 leader_election.cc:304] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:14.300875 6224 raft_consensus.cc:2410] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:14.295514 6306 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:14.301425 6273 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:14.304973 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 637) took 2830 ms. Trace:
I20260620 19:59:14.296847 8708 consensus_queue.cc:1059] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260620 19:59:14.300117 8901 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.305337 6306 rpcz_store.cc:276] 0620 19:59:11.474437 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.294764 (+2820327us) service_pool.cc:224] Handling call
0620 19:59:14.295109 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.304739 (+ 9630us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.304921 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.319100 6226 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:14.319937 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.302551 6230 raft_consensus.cc:2393] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:14.302997 6304 rpcz_store.cc:276] 0620 19:59:12.503970 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.234290 (+1730320us) service_pool.cc:224] Handling call
0620 19:59:14.264424 (+ 30134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.338275 6224 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:14.339618 6224 raft_consensus.cc:1240] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:14.340508 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.340876 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 638) took 2866 ms. Trace:
I20260620 19:59:14.341260 6299 rpcz_store.cc:276] 0620 19:59:11.474811 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.298269 (+2823458us) service_pool.cc:224] Handling call
0620 19:59:14.340415 (+ 42146us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.340576 (+ 161us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:14.340600 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:14.340639 (+ 39us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:14.340674 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.340729 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:14.340828 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.276412 8815 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260620 19:59:14.301366 6300 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:14.296306 8827 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:14.293978 6298 raft_consensus.cc:2376] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7. Re-sending same reply.
I20260620 19:59:14.350890 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.352705 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1569) took 1163 ms. Trace:
I20260620 19:59:14.353106 6303 rpcz_store.cc:276] 0620 19:59:13.189044 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.350633 (+1161589us) service_pool.cc:224] Handling call
0620 19:59:14.352640 (+ 2007us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.300434 8618 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260620 19:59:14.342317 6224 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260620 19:59:14.315215 8779 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:14.314738 8813 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:14.291715 6902 rpcz_store.cc:276] 0620 19:59:13.233194 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.175805 (+942611us) service_pool.cc:224] Handling call
0620 19:59:14.261346 (+ 85541us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.419574 8813 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.420034 8787 consensus_queue.cc:1048] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:14.420256 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.421641 8813 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.422140 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1835) took 1108 ms. Trace:
I20260620 19:59:14.421973 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.423295 6299 raft_consensus.cc:1240] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:14.355872 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.418957 8779 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:14.424111 6301 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:14.424369 6225 raft_consensus.cc:1240] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260620 19:59:14.425731 6303 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:14.325330 6905 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEARNER]: Advancing to term 6
I20260620 19:59:14.308228 8730 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:14.431704 8730 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:14.432258 8730 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:14.428225 6230 raft_consensus.cc:1240] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260620 19:59:14.428670 6907 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:14.436368 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.305485 8906 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:14.437613 8906 raft_consensus.cc:697] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:14.439213 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 853) took 1269 ms. Trace:
I20260620 19:59:14.438825 8906 consensus_queue.cc:237] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:14.439631 6910 rpcz_store.cc:276] 0620 19:59:13.169695 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.211803 (+1042108us) service_pool.cc:224] Handling call
0620 19:59:14.220170 (+ 8367us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.220365 (+ 195us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:14.241624 (+ 21259us) log.cc:844] Serialized 14 byte log entry
0620 19:59:14.438883 (+197259us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:14.438943 (+ 60us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:14.438993 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.439051 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:14.439163 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20020,"prepare.queue_time_us":54201,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":130337,"threads_started":2}
I20260620 19:59:14.311909 8809 raft_consensus.cc:2804] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:14.429733 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1780) took 2691 ms. Trace:
I20260620 19:59:14.441432 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.441731 6303 rpcz_store.cc:276] 0620 19:59:11.734699 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.425287 (+2690588us) service_pool.cc:224] Handling call
0620 19:59:14.425516 (+ 229us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.426197 (+ 681us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.426409 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.442142 6910 raft_consensus.cc:2393] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 6: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:14.445693 6303 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:14.306402 8842 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260620 19:59:14.457341 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 293444us
I20260620 19:59:14.430692 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1777) took 2793 ms. Trace:
I20260620 19:59:14.459949 6224 raft_consensus.cc:1240] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:14.460021 6299 rpcz_store.cc:276] 0620 19:59:11.635055 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.342110 (+2707055us) service_pool.cc:224] Handling call
0620 19:59:14.423116 (+ 81006us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.423728 (+ 612us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.428185 (+ 4457us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.314463 8901 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:14.346525 8753 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.348553 8776 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.350731 8798 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:14.456794 6304 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:14.487690 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1558) took 2338 ms. Trace:
I20260620 19:59:14.488164 6304 rpcz_store.cc:276] 0620 19:59:12.149534 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.443619 (+2294085us) service_pool.cc:224] Handling call
0620 19:59:14.443911 (+ 292us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.487335 (+ 43424us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.487619 (+ 284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.489714 6304 raft_consensus.cc:1240] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:14.490412 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1789) took 2213 ms. Trace:
I20260620 19:59:14.494737 6304 rpcz_store.cc:276] 0620 19:59:12.277247 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.489099 (+2211852us) service_pool.cc:224] Handling call
0620 19:59:14.489471 (+ 372us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.490147 (+ 676us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.490353 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.306799 8728 raft_consensus.cc:2804] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:14.458961 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1787) took 2282 ms. Trace:
I20260620 19:59:14.502010 6303 rpcz_store.cc:276] 0620 19:59:12.164246 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.444860 (+2280614us) service_pool.cc:224] Handling call
0620 19:59:14.445315 (+ 455us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.446125 (+ 810us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.446360 (+ 235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.503788 6303 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260620 19:59:14.504451 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1808) took 1695 ms. Trace:
I20260620 19:59:14.504861 6303 rpcz_store.cc:276] 0620 19:59:12.809032 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.503027 (+1693995us) service_pool.cc:224] Handling call
0620 19:59:14.503383 (+ 356us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.504175 (+ 792us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.504391 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.308071 8864 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.465574 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.467068 6299 raft_consensus.cc:1240] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 5. Ops: []
I20260620 19:59:14.513952 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1788) took 2301 ms. Trace:
I20260620 19:59:14.467466 8901 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.475868 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.515161 6304 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260620 19:59:14.427744 8779 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:14.422623 6298 rpcz_store.cc:276] 0620 19:59:13.313378 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.208137 (+894759us) service_pool.cc:224] Handling call
0620 19:59:14.422077 (+213940us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.515251 6231 raft_consensus.cc:2393] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:14.302632 8781 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:14.514400 6299 rpcz_store.cc:276] 0620 19:59:12.212633 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.460890 (+2248257us) service_pool.cc:224] Handling call
0620 19:59:14.466902 (+ 6012us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.513657 (+ 46755us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.513887 (+ 230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.516319 6304 pending_rounds.cc:85] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 4
I20260620 19:59:14.520123 6304 pending_rounds.cc:107] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260620 19:59:14.517370 6303 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:14.520463 6304 raft_consensus.cc:2889] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:14.522432 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1820) took 1555 ms. Trace:
I20260620 19:59:14.524164 6303 rpcz_store.cc:276] 0620 19:59:12.967537 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.516649 (+1549112us) service_pool.cc:224] Handling call
0620 19:59:14.517010 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.521283 (+ 4273us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.522379 (+ 1096us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.526823 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1799) took 2008 ms. Trace:
I20260620 19:59:14.527280 6304 rpcz_store.cc:276] 0620 19:59:12.518368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.514184 (+1995816us) service_pool.cc:224] Handling call
0620 19:59:14.514560 (+ 376us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.526577 (+ 12017us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.526770 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.528404 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1564) took 1363 ms. Trace:
I20260620 19:59:14.528887 6299 rpcz_store.cc:276] 0620 19:59:13.164640 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.518848 (+1354208us) service_pool.cc:224] Handling call
0620 19:59:14.527807 (+ 8959us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.528007 (+ 200us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:14.528043 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:14.528098 (+ 55us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:14.528141 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.528223 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:14.528344 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:14.522531 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 216768us
I20260620 19:59:14.530369 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:14.531664 6226 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:14.532477 6907 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:14.534960 6303 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:14.539491 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1567) took 1351 ms. Trace:
I20260620 19:59:14.541050 6303 rpcz_store.cc:276] 0620 19:59:13.188427 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.526167 (+1337740us) service_pool.cc:224] Handling call
0620 19:59:14.534619 (+ 8452us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.539251 (+ 4632us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.539435 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.539958 6874 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:14.551306 6298 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260620 19:59:14.458357 8833 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260620 19:59:14.554443 6299 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:14.558749 6905 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260620 19:59:14.640038 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 861) took 1387 ms. Trace:
I20260620 19:59:14.640468 6905 rpcz_store.cc:276] 0620 19:59:13.252771 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.261673 (+1008902us) service_pool.cc:224] Handling call
0620 19:59:14.303456 (+ 41783us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.639681 (+336225us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.639967 (+ 286us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:14.562198 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 257547us
I20260620 19:59:14.508021 8753 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.507865 8776 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.514605 8691 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:14.531036 8728 raft_consensus.cc:697] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [term 7 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:14.580116 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1571) took 1346 ms. Trace:
I20260620 19:59:14.639063 6298 pending_rounds.cc:85] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 3
I20260620 19:59:14.580045 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:14.638019 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.802858 6196 leader_election.cc:336] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 1.596s (SENT)
W20260620 19:59:14.864627 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 61825.6us
I20260620 19:59:14.570588 8708 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.815680 6304 rpcz_store.cc:276] 0620 19:59:13.189560 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.529009 (+1339449us) service_pool.cc:224] Handling call
0620 19:59:14.529345 (+ 336us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.529548 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:14.529583 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:14.529636 (+ 53us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:14.529686 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.535461 (+ 5775us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:14.535570 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.865761 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.866919 6905 raft_consensus.cc:2410] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260620 19:59:14.867246 6304 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:14.513352 8798 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:14.832109 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:14.887372 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 652) took 1402 ms. Trace:
W20260620 19:59:14.888082 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:14.559623 6223 raft_consensus.cc:1240] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260620 19:59:14.858863 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:14.862165 6298 pending_rounds.cc:107] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260620 19:59:14.830910 8728 consensus_queue.cc:237] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.861521 6303 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:14.900209 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1836) took 1543 ms. Trace:
I20260620 19:59:14.900511 6303 rpcz_store.cc:276] 0620 19:59:13.356463 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.584155 (+1227692us) service_pool.cc:224] Handling call
0620 19:59:14.861088 (+276933us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.899984 (+ 38896us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.900159 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:14.870805 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.901574 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.864318 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 402291us
I20260620 19:59:14.902812 6908 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260620 19:59:14.903113 8916 log.cc:927] Time spent T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.320s user 0.000s sys 0.000s
I20260620 19:59:14.904152 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.904769 6902 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:14.905651 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.906167 6902 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:14.879092 6344 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.033s sys 0.068s
W20260620 19:59:14.914824 6344 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.545s user 0.033s sys 0.069s
I20260620 19:59:14.887750 6304 rpcz_store.cc:276] 0620 19:59:13.485581 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.866486 (+1380905us) service_pool.cc:224] Handling call
0620 19:59:14.866864 (+ 378us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.867684 (+ 820us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.887309 (+ 19625us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:14.858690 8776 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:14.893627 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1829) took 1641 ms. Trace:
I20260620 19:59:14.895921 8708 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.915704 6299 rpcz_store.cc:276] 0620 19:59:13.241036 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.529977 (+1288941us) service_pool.cc:224] Handling call
0620 19:59:14.554053 (+ 24076us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.639339 (+ 85286us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.869549 (+230210us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:14.899360 6903 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 670) took 2062 ms (client timeout 1596 ms). Trace:
I20260620 19:59:14.909533 6911 raft_consensus.cc:1240] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:14.917091 8708 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:14.917572 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 859) took 1682 ms. Trace:
I20260620 19:59:14.917650 8798 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.917903 6911 rpcz_store.cc:276] 0620 19:59:13.235050 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.259182 (+1024132us) service_pool.cc:224] Handling call
0620 19:59:14.909046 (+649864us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:14.909284 (+ 238us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.917345 (+ 8061us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.917523 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8728320}
I20260620 19:59:14.919575 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:14.920288 6911 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:14.868105 8920 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260620 19:59:14.916741 6903 rpcz_store.cc:269] 0620 19:59:12.836515 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.836655 (+ 140us) service_pool.cc:224] Handling call
0620 19:59:14.899096 (+2062441us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54005014d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.899257 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":358766848}
I20260620 19:59:14.920945 8920 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:14.921307 8920 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:14.922160 6903 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:14.923628 6903 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260620 19:59:14.924826 6903 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:14.926008 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 678) took 2082 ms. Trace:
I20260620 19:59:14.926436 6906 rpcz_store.cc:276] 0620 19:59:12.843499 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:13.090350 (+246851us) service_pool.cc:224] Handling call
0620 19:59:13.267022 (+176672us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:13.290997 (+ 23975us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.925790 (+1634793us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540052c450. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.925933 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":399601536}
I20260620 19:59:14.928129 8753 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.931166 8618 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.931754 8618 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.932494 8901 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.933807 8618 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:14.934082 8781 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.935634 8618 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:14.867987 8691 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:14.914717 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:14.934540 8781 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:14.936538 6905 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:14.938362 8781 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:14.938470 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 487) took 1595 ms. Trace:
I20260620 19:59:14.938921 6374 rpcz_store.cc:276] 0620 19:59:13.343089 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:13.343191 (+ 102us) service_pool.cc:224] Handling call
0620 19:59:13.348154 (+ 4963us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.148619 (+800465us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.938224 (+789605us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b5400064d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:14.938391 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1838631808}
I20260620 19:59:14.940951 8618 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:14.941782 8808 consensus_meta.cc:287] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.694s user 0.005s sys 0.001s
W20260620 19:59:14.942000 6301 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1784) took 2924 ms (client timeout 1765 ms). Trace:
W20260620 19:59:14.942509 6301 rpcz_store.cc:269] 0620 19:59:12.018097 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.219091 (+200994us) service_pool.cc:224] Handling call
0620 19:59:14.941834 (+2722743us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54003eeed4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.941934 (+ 100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5029349248}
W20260620 19:59:14.945199 6228 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1916) took 1434 ms (client timeout 1763 ms). Trace:
I20260620 19:59:14.945634 8618 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:14.945942 8912 consensus_meta.cc:287] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.757s user 0.005s sys 0.003s
I20260620 19:59:14.929989 8779 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:14.947166 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 577316us
I20260620 19:59:14.886334 6907 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:14.911470 6226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1909) took 1422 ms. Trace:
I20260620 19:59:14.947582 6905 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
W20260620 19:59:14.948104 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 460596us
W20260620 19:59:14.948601 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 418229us
W20260620 19:59:14.945600 6228 rpcz_store.cc:269] 0620 19:59:13.510453 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.048240 (+537787us) service_pool.cc:224] Handling call
0620 19:59:14.945019 (+896779us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b5400077954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.945143 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61240576}
I20260620 19:59:14.952960 6874 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:14.951565 6226 rpcz_store.cc:276] 0620 19:59:13.489350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.337448 (+848098us) service_pool.cc:224] Handling call
0620 19:59:14.351930 (+ 14482us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.532137 (+180207us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.911249 (+379112us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54001e41d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:14.911414 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":412816768}
W20260620 19:59:14.951171 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.947274 8730 consensus_meta.cc:287] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.000s
W20260620 19:59:14.954465 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:14.955260 6874 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
W20260620 19:59:14.956316 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.959005 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:14.959658 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:14.959867 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:14.960846 6377 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:14.961287 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.962006 6302 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:14.942195 8808 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:14.963719 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.946785 6298 raft_consensus.cc:2889] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:14.965197 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.966976 6303 raft_consensus.cc:1275] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:14.889585 8827 raft_consensus.cc:697] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:14.968225 6302 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:14.976751 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:14.977542 6302 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:14.978549 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.979509 6302 raft_consensus.cc:2376] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9. Re-sending same reply.
I20260620 19:59:14.980511 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:14.981113 6302 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:14.984340 6307 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1812) took 2174 ms (client timeout 2009 ms). Trace:
W20260620 19:59:14.984728 6307 rpcz_store.cc:269] 0620 19:59:12.810158 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:12.850312 (+ 40154us) service_pool.cc:224] Handling call
0620 19:59:14.984156 (+2133844us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b54003250d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:14.984267 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1412664704}
I20260620 19:59:14.986296 6307 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260620 19:59:14.950220 8809 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.003s sys 0.007s
W20260620 19:59:14.989485 8809 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.004s sys 0.007s
I20260620 19:59:15.000442 6374 raft_consensus.cc:1275] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260620 19:59:14.967796 8901 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:14.970398 8776 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:14.971920 8730 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.011128 8643 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.010921 8920 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.954128 8912 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:14.976667 8798 consensus_queue.cc:1059] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260620 19:59:14.970060 8751 consensus_queue.cc:1048] T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260620 19:59:15.002846 8809 raft_consensus.cc:697] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:15.003432 8813 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260620 19:59:15.008050 8901 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:15.022070 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 690) took 2032 ms. Trace:
I20260620 19:59:15.035792 6908 rpcz_store.cc:276] 0620 19:59:12.989722 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.197502 (+1207780us) service_pool.cc:224] Handling call
0620 19:59:14.902391 (+704889us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:14.903323 (+ 932us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:15.021875 (+118552us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b54005014d0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:15.022007 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1619911936}
I20260620 19:59:15.017469 8776 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:15.036829 8776 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:15.036419 8809 consensus_queue.cc:237] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.035792 8901 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:15.043668 8779 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.044354 8912 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:15.012672 6904 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 697) took 1796 ms (client timeout 1596 ms). Trace:
W20260620 19:59:15.045434 6904 rpcz_store.cc:269] 0620 19:59:13.207944 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.994952 (+1787008us) service_pool.cc:211] Skipping call since client already timed out
0620 19:59:15.004250 (+ 9298us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260620 19:59:15.047026 8643 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:14.976794 8753 consensus_queue.cc:1059] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260620 19:59:15.034024 8864 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:15.035146 8808 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.048573 8730 leader_election.cc:290] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:14.968825 8618 consensus_queue.cc:1059] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260620 19:59:15.094107 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:15.023566 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 65876.5us
I20260620 19:59:15.136170 8808 raft_consensus.cc:3055] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:15.136786 8808 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:15.137732 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 44204.1us
I20260620 19:59:15.135509 8618 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:15.106952 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:15.108752 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:14.963975 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:15.145841 6299 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:15.147470 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.148227 6299 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.148943 8920 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.150087 8918 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:14.971131 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:15.112737 6300 raft_consensus.cc:1240] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 3. Current term is 5. Ops: []
I20260620 19:59:15.113090 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1566) took 1947 ms. Trace:
I20260620 19:59:15.151427 6304 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:15.151134 6298 rpcz_store.cc:276] 0620 19:59:13.188141 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.522315 (+1334174us) service_pool.cc:224] Handling call
0620 19:59:14.550966 (+ 28651us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:15.112356 (+561390us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:15.112844 (+ 488us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:15.113370 6301 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:15.152861 6344 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.153461 6304 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:15.153510 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1779) took 3462 ms. Trace:
I20260620 19:59:15.143699 8881 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260620 19:59:15.154759 6301 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:14.982736 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:15.155792 6302 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:15.157318 8934 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 19:59:15.157895 8934 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:15.157809 8901 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:15.002123 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 69922.7us
I20260620 19:59:15.123534 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:15.125541 6223 raft_consensus.cc:1275] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260620 19:59:15.127457 6228 raft_consensus.cc:1275] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260620 19:59:15.127461 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.128692 6373 raft_consensus.cc:1240] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:15.129889 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.130896 6377 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:15.131095 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.131871 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.132216 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.132947 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.129884 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.139520 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:15.184818 6378 raft_consensus.cc:2393] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:15.138633 8808 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.060235 8827 consensus_queue.cc:237] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.187295 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.188212 6380 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:15.141733 8829 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:15.052297 8643 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:15.103709 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 394) took 1016 ms. Trace:
I20260620 19:59:15.202988 6225 rpcz_store.cc:276] 0620 19:59:14.087506 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.444161 (+356655us) service_pool.cc:224] Handling call
0620 19:59:14.468348 (+ 24187us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:14.468673 (+ 325us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:14.468700 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:14.469165 (+ 465us) log.cc:844] Serialized 26 byte log entry
0620 19:59:14.893192 (+424027us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:14.893238 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:14.893333 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:15.102834 (+209501us) raft_consensus.cc:1685] finished
0620 19:59:15.102929 (+ 95us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:15.103406 (+ 477us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":996,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":35675,"thread_start_us":423613,"threads_started":1,"wal-append.queue_time_us":521801}
I20260620 19:59:15.158423 8934 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:15.204322 8643 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.116392 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.158834 8906 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:59:15.146426 8882 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260620 19:59:15.160504 8618 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.207172 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:15.215152 8618 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:15.051152 8814 raft_consensus.cc:3055] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:15.216275 8814 raft_consensus.cc:740] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:15.217238 6129 catalog_manager.cc:5696] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:15.154258 6300 rpcz_store.cc:276] 0620 19:59:11.691420 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.344532 (+2653112us) service_pool.cc:224] Handling call
0620 19:59:15.112011 (+767479us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:15.151137 (+ 39126us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:15.153274 (+ 2137us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b54003250d0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:15.153439 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1578171648}
I20260620 19:59:15.219647 6129 catalog_manager.cc:5696] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:59:15.164141 8813 raft_consensus.cc:3055] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:15.220796 8813 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:15.220896 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:15.221612 6228 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:15.222143 8813 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.223750 8813 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:15.218221 8814 consensus_queue.cc:260] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.164368 8937 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.225117 8814 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:15.225569 8937 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.226749 6904 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:15.228051 8937 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:15.164719 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:15.165236 8756 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.229622 8756 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.170846 8918 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:15.171975 6306 raft_consensus.cc:1240] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 8. Ops: []
I20260620 19:59:15.230820 8918 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:15.178442 6345 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:15.232239 8935 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:15.232748 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1776) took 3689 ms. Trace:
I20260620 19:59:15.233174 6306 rpcz_store.cc:276] 0620 19:59:11.543661 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:14.318997 (+2775336us) service_pool.cc:224] Handling call
0620 19:59:14.319387 (+ 390us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:15.230954 (+911567us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:15.232490 (+ 1536us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b5400322dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:15.232690 (+ 200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1959282944}
I20260620 19:59:15.173115 6298 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
W20260620 19:59:15.233839 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 55682.7us
I20260620 19:59:15.177249 8926 consensus_queue.cc:1059] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:15.179692 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.235370 8926 raft_consensus.cc:3055] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:15.235809 8926 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:15.236534 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.237027 8926 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.163404 6874 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.180877 6299 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:15.181296 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.239200 6307 raft_consensus.cc:1240] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260620 19:59:15.239372 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 76337.9us
I20260620 19:59:15.239727 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.240659 8798 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:15.240638 6874 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.241153 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.183310 8781 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:15.242015 8753 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:15.185833 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.242818 6902 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:15.243299 6378 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 5
I20260620 19:59:15.243866 8753 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:15.243927 8756 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.244441 8753 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:15.244936 8753 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.189211 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.246958 8926 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:15.247570 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:15.248735 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.248900 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.182633 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.179795 8901 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:15.242136 6874 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.166050 6344 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:15.194877 6381 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:15.141186 6195 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:15.196548 6379 raft_consensus.cc:2368] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:59:15.197088 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.200399 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:15.201851 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:15.187295 8808 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:15.241284 8798 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:15.257457 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.195825 6375 raft_consensus.cc:2368] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7.
I20260620 19:59:15.250260 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.238319 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.182649 6304 raft_consensus.cc:1275] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260620 19:59:15.263079 8751 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.263302 6271 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:15.263716 8751 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.251130 6129 catalog_manager.cc:5696] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 4 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:59:15.264933 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 123976us
I20260620 19:59:15.265450 6271 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:15.266834 8618 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:59:15.266379 6129 catalog_manager.cc:5696] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:59:15.267824 6874 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.268934 6380 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:15.268270 6129 catalog_manager.cc:5696] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:59:15.269228 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 103387us
I20260620 19:59:15.270910 6344 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:15.271726 6381 raft_consensus.cc:1240] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260620 19:59:15.271859 8935 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:15.274655 8809 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:15.275357 8809 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.276049 6375 raft_consensus.cc:1240] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:15.266856 6377 raft_consensus.cc:3060] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:15.277040 8776 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.267724 6379 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260620 19:59:15.282976 8901 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:15.283210 8935 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:15.283371 8901 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:15.283807 8901 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.267674 8618 consensus_queue.cc:1048] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:15.266346 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:15.270136 6129 catalog_manager.cc:5696] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 6, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 6 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:15.279131 6373 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:15.289345 6228 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 10.
I20260620 19:59:15.300769 8618 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:15.301159 8618 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:15.294039 8756 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:15.294661 6373 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260620 19:59:15.295058 8779 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:15.296161 8928 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:15.258666 8798 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:15.302030 8756 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:15.303257 8756 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.307179 6378 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:15.297065 6129 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:15.299371 6195 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:15.301600 8618 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:15.266489 8751 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:15.313153 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 78339.8us
I20260620 19:59:15.277626 8809 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:15.319797 6381 raft_consensus.cc:1275] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260620 19:59:15.319797 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.278048 8934 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.330761 8882 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:15.334519 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:15.340811 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:15.347988 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 82712.9us
I20260620 19:59:15.349459 6373 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:15.347512 8882 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:15.360328 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 86842.6us
I20260620 19:59:15.360713 8882 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.362830 8781 consensus_queue.cc:1059] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260620 19:59:15.364467 8781 raft_consensus.cc:3055] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:15.364749 6345 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:15.364943 8781 raft_consensus.cc:740] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:15.366139 8915 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:15.371596 8829 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:15.372108 6195 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:15.373895 8829 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:59:15.306530 8937 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260620 19:59:15.374383 8730 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:15.374912 8730 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:15.375397 8730 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:15.374997 8864 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:15.376008 8937 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.376737 8937 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.377091 8864 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:15.364984 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:15.379355 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 67869us
I20260620 19:59:15.379889 6381 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:15.381889 8865 consensus_queue.cc:1059] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:15.383219 8865 raft_consensus.cc:3055] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:15.383585 8865 raft_consensus.cc:740] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:15.384682 8776 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:15.384572 8865 consensus_queue.cc:260] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.385951 8865 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.388437 8934 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:15.389707 8643 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.366343 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.392249 6374 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:15.394503 8618 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.396504 8753 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.397781 6907 raft_consensus.cc:2393] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:15.399340 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:15.399904 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.399760 8756 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:15.400467 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.396943 8618 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.404798 8798 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.406529 8776 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.406860 6271 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:15.407306 8776 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.407775 8814 raft_consensus.cc:2764] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260620 19:59:15.409222 6271 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:15.409602 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.410106 8814 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:15.410741 8776 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:15.413048 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 46861.6us
I20260620 19:59:15.417640 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:15.412473 8756 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.419571 8921 consensus_queue.cc:1059] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:15.420113 8935 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:59:15.420506 8915 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:15.419660 8934 consensus_queue.cc:1059] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:15.422395 8934 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:15.423777 8934 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:15.424813 8934 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.426079 8934 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:15.426545 8915 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:15.427121 8918 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.427273 8730 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.429121 8918 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:15.429704 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 73596.3us
I20260620 19:59:15.431022 8753 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:15.431990 6273 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.432922 8901 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.433421 8814 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:15.421571 6344 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.435029 8814 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:15.436721 6377 raft_consensus.cc:2468] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 10.
I20260620 19:59:15.435475 8814 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:15.446945 8814 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:15.452556 8915 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election won for term 7
W20260620 19:59:15.453200 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 141932us
I20260620 19:59:15.455193 6273 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.454855 6195 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:15.456693 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 82059.3us
I20260620 19:59:15.459126 8781 consensus_queue.cc:260] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.459337 6198 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:15.460300 8779 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:59:15.459803 8751 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:15.460582 8781 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:15.456720 6273 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.456610 8730 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:15.465744 8751 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:15.502712 8926 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260620 19:59:15.504657 8926 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:15.485322 8865 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:15.490520 6195 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:15.499115 8751 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:15.508334 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 44905.1us
I20260620 19:59:15.508708 6875 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:15.510082 6875 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:15.514155 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:15.515406 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 80707.8us
I20260620 19:59:15.510867 8730 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:15.495692 8937 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.520277 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.499274 8918 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:15.500777 8808 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
W20260620 19:59:15.534731 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.535182 8901 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.536262 6299 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:15.539625 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 104082us
I20260620 19:59:15.512779 8921 consensus_queue.cc:1243] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260620 19:59:15.535830 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.541903 6301 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.543287 6345 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.544237 6345 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.539467 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:15.545087 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 76024.5us
I20260620 19:59:15.550182 8912 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:15.551241 8912 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:15.551652 8912 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:15.553228 8814 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.494132 8882 consensus_queue.cc:1059] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20260620 19:59:15.564925 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.568478 6875 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:15.532852 8928 raft_consensus.cc:2804] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader election won for term 10
W20260620 19:59:15.570535 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.570755 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.538221 8813 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:15.573937 8901 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.516171 8618 consensus_queue.cc:1059] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260620 19:59:15.574489 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.575372 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.564493 8808 raft_consensus.cc:697] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:15.575940 8921 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:15.576882 6128 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47072 (request call id 395) took 8758 ms. Trace:
I20260620 19:59:15.622267 6128 rpcz_store.cc:276] 0620 19:59:06.818063 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:06.874893 (+ 56830us) service_pool.cc:224] Handling call
0620 19:59:15.567957 (+8693064us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:59:11.145305 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:59:11.421200 (+275895us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:59:11.458921 (+ 37721us) write_op.cc:435] Acquired schema lock
0620 19:59:11.464654 (+ 5733us) tablet.cc:662] Decoding operations
0620 19:59:12.059268 (+594614us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:59:12.066858 (+ 7590us) write_op.cc:641] Partition lock acquired for write op
0620 19:59:12.069386 (+ 2528us) tablet.cc:685] Acquiring locks for 13 operations
0620 19:59:12.136642 (+ 67256us) tablet.cc:701] Row locks acquired
0620 19:59:12.136675 (+ 33us) write_op.cc:260] PREPARE: finished
0620 19:59:12.136870 (+ 195us) write_op.cc:270] Start()
0620 19:59:12.137017 (+ 147us) write_op.cc:276] Timestamp: P: 1781985552136796 usec, L: 0
0620 19:59:12.137038 (+ 21us) op_driver.cc:348] REPLICATION: starting
0620 19:59:12.137719 (+ 681us) log.cc:844] Serialized 5213 byte log entry
0620 19:59:12.515039 (+377320us) op_driver.cc:464] REPLICATION: finished
0620 19:59:12.794377 (+279338us) write_op.cc:301] APPLY: starting
0620 19:59:12.794473 (+ 96us) tablet.cc:1367] starting BulkCheckPresence
0620 19:59:12.794797 (+ 324us) tablet.cc:1370] finished BulkCheckPresence
0620 19:59:12.794943 (+ 146us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:59:12.796469 (+ 1526us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:59:12.796759 (+ 290us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0620 19:59:12.796810 (+ 51us) write_op.cc:312] APPLY: finished
0620 19:59:12.797176 (+ 366us) log.cc:844] Serialized 95 byte log entry
0620 19:59:12.798275 (+ 1099us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:12.798435 (+ 160us) write_op.cc:454] Released schema lock
0620 19:59:12.798492 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0620 19:59:12.798921 (+ 429us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:12.798956 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":601345,"reported_tablets":34,"child_traces":[["op",{"apply.queue_time_us":272304,"apply.run_cpu_time_us":3615,"apply.run_wall_time_us":6782,"num_ops":13,"prepare.queue_time_us":339278,"prepare.run_cpu_time_us":47798,"prepare.run_wall_time_us":1125369,"raft.queue_time_us":146799,"raft.run_cpu_time_us":70038,"raft.run_wall_time_us":180569,"replication_time_us":377832,"thread_start_us":643844,"threads_started":4,"wal-append.queue_time_us":202452,"wal-append.run_cpu_time_us":13354,"wal-append.run_wall_time_us":1454427}]]}
I20260620 19:59:15.537433 8730 raft_consensus.cc:697] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:15.627444 8730 consensus_queue.cc:237] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:15.577850 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 127390us
I20260620 19:59:15.536288 8776 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:15.580297 6379 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:15.620093 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 75915us
I20260620 19:59:15.573261 6306 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:15.539382 8915 raft_consensus.cc:697] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:59:15.604895 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.614657 6301 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:15.533221 8865 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260620 19:59:15.616281 8882 consensus_queue.cc:1243] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260620 19:59:15.631115 6198 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:15.635763 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:15.642366 6378 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260620 19:59:15.647179 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 43569.4us
I20260620 19:59:15.640187 6379 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:15.642058 6874 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.644452 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.656694 8954 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:15.657871 8961 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:15.659334 8961 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:15.659585 8955 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:15.660434 8954 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:15.662671 8961 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:15.663170 8954 raft_consensus.cc:697] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:15.664431 8954 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.662109 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.669871 6303 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:15.635396 6345 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.617902 8813 raft_consensus.cc:697] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:15.677940 6874 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.672351 6305 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:15.682389 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 47467.1us
W20260620 19:59:15.683151 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 153096us
I20260620 19:59:15.621045 8928 raft_consensus.cc:697] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [term 10 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:15.630913 8776 raft_consensus.cc:697] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:15.755290 8776 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.654120 8798 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:15.697351 6299 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:15.678921 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.635929 8915 consensus_queue.cc:237] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:15.682308 8756 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:15.630255 8901 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.700104 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:15.682480 6298 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:15.704892 8813 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.705741 6306 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:15.706434 8787 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:15.707901 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.778221 6302 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:15.710820 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 79823.9us
I20260620 19:59:15.711367 6129 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 7 to 9, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 9 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:15.713665 8935 raft_consensus.cc:3055] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:15.714159 8928 consensus_queue.cc:237] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:15.715378 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.715374 8827 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:15.675203 8865 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.720160 8864 raft_consensus.cc:3055] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:15.727777 8814 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:15.784607 8827 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:15.784788 8865 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.785462 8814 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:59:15.786026 8814 raft_consensus.cc:697] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:15.677044 8937 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:15.786790 8937 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:15.787215 8937 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:15.680823 8618 consensus_queue.cc:1243] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260620 19:59:15.743304 8953 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:15.788861 8953 raft_consensus.cc:697] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:15.789950 8953 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.791339 8829 raft_consensus.cc:3055] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:15.787220 8865 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:15.736447 6875 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:15.748351 6301 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:15.743844 6345 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.676892 8643 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260620 19:59:15.769724 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 90905.1us
I20260620 19:59:15.770924 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:15.771889 8756 raft_consensus.cc:697] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:59:15.787317 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.791021 8814 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:15.747478 8949 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260620 19:59:15.785276 8827 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:15.811380 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 115614us
I20260620 19:59:15.793325 6305 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:15.742391 8751 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.764746 8918 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:15.807928 6874 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.776861 8809 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:15.791867 8829 raft_consensus.cc:740] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:15.813828 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:15.815428 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 250438us
I20260620 19:59:15.784727 8864 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:15.803341 8643 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:15.821563 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.779783 8935 raft_consensus.cc:740] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:15.805876 8798 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:59:15.838328 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:15.839334 6304 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:15.841228 6378 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:15.842638 6874 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:15.823627 8926 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:15.758169 6303 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:15.851287 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.826658 8643 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:15.839888 8781 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:15.814422 8912 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.814728 8808 consensus_queue.cc:237] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:15.869712 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 133359us
I20260620 19:59:15.842490 8921 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260620 19:59:15.868884 8920 raft_consensus.cc:2804] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election won for term 6
W20260620 19:59:15.880398 8956 log.cc:927] Time spent T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.177s user 0.002s sys 0.000s
I20260620 19:59:15.842530 8865 consensus_queue.cc:1059] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260620 19:59:15.881402 8901 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:15.881006 8935 consensus_queue.cc:260] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:15.882656 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.882172 8829 consensus_queue.cc:260] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:15.891083 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 191075us
I20260620 19:59:15.866797 8926 raft_consensus.cc:697] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:59:15.900321 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.903311 6302 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:15.885185 8798 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:15.915201 6378 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:15.884019 6129 catalog_manager.cc:5696] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } }
W20260620 19:59:15.917268 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 74708.6us
I20260620 19:59:15.924172 6271 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:15.925590 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:15.929191 6301 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:15.901245 8920 raft_consensus.cc:697] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:59:15.930651 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 85065us
I20260620 19:59:15.883582 8781 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.920997 8798 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:15.935737 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:15.936134 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 111464us
I20260620 19:59:15.933130 6129 catalog_manager.cc:5696] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:59:15.937145 8809 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:15.938130 8798 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:15.958746 6129 catalog_manager.cc:5696] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 4 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:15.908670 8756 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:15.972016 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 47932.1us
I20260620 19:59:15.946661 8829 raft_consensus.cc:3060] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:15.977291 6273 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:15.957115 8935 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:15.955197 8958 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.131s
I20260620 19:59:15.985396 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:15.966610 8751 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:15.933843 8918 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.990975 6129 catalog_manager.cc:5696] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 3, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 3 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:15.935185 8864 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:16.001191 6345 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:16.006623 6129 catalog_manager.cc:5696] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 5 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:59:16.008910 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 64486us
W20260620 19:59:16.111764 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.108580 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 107009us
W20260620 19:59:16.112195 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 108231us
W20260620 19:59:16.083351 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 104869us
W20260620 19:59:16.043799 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 64792us
W20260620 19:59:16.012969 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.141634 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.141817 6874 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:16.146394 6874 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:16.147459 8901 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:16.148867 6195 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:15.980284 8926 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.095569 6129 catalog_manager.cc:5696] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } }
W20260620 19:59:16.154425 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.156801 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.144474 8882 raft_consensus.cc:3055] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
W20260620 19:59:16.086324 8964 log.cc:927] Time spent T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
I20260620 19:59:16.168344 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:16.177516 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:16.178197 6378 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:16.012559 8954 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.012406 8921 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:16.121336 8776 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.160346 8730 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.120047 6298 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:16.178884 6271 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:16.194921 6302 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260620 19:59:16.111364 8972 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:16.156594 8864 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:16.042418 8912 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:16.172086 8865 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:16.167420 8882 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:16.199736 6345 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:16.218755 8901 raft_consensus.cc:697] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:16.215739 6129 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 4 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 5 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:59:16.239773 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.163880 8961 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:15.998716 8809 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:16.249488 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 81816.7us
W20260620 19:59:16.250736 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.206984 8781 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:16.258332 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.270151 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.219892 8954 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.235833 8618 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:16.086796 8920 consensus_queue.cc:237] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:16.281529 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 100637us
I20260620 19:59:16.277441 8882 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:16.281837 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 103042us
I20260620 19:59:16.277354 8973 raft_consensus.cc:2804] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:16.276726 8975 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:16.236038 8865 raft_consensus.cc:697] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:16.249576 8953 raft_consensus.cc:2764] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260620 19:59:16.292140 8948 log.cc:927] Time spent T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260620 19:59:16.360998 6129 catalog_manager.cc:5696] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 3 to 6, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131). New cstate: current_term: 6 leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:59:16.215348 8730 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:16.375008 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 208859us
W20260620 19:59:16.379093 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 206717us
I20260620 19:59:16.249186 8776 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:16.372179 6299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.019s sys 0.003s
I20260620 19:59:16.366220 8618 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:16.400863 8976 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:16.401134 8865 consensus_queue.cc:237] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.401381 8618 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:16.401933 8961 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:16.394758 8973 raft_consensus.cc:697] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:16.406430 8973 consensus_queue.cc:237] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:16.425496 6129 catalog_manager.cc:5696] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 1 to 2, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 2 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:59:16.438560 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.400120 6299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.531s user 0.020s sys 0.003s
W20260620 19:59:16.447131 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 72818us
I20260620 19:59:16.409286 8975 raft_consensus.cc:697] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:16.428436 8808 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.475474 6298 raft_consensus.cc:1240] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260620 19:59:16.481456 6378 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
W20260620 19:59:16.472086 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 133545us
I20260620 19:59:16.340632 8901 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.488843 6129 catalog_manager.cc:5696] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 reported cstate change: term changed from 2 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 5 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:16.439486 8976 raft_consensus.cc:697] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:16.504022 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.493438 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 152881us
I20260620 19:59:16.508572 6198 leader_election.cc:400] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:59:16.488241 8882 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:16.509044 6198 leader_election.cc:403] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:59:16.514256 8990 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.515661 6907 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
W20260620 19:59:16.519176 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.530366 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 119138us
W20260620 19:59:16.532334 8979 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
I20260620 19:59:16.532696 8993 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
W20260620 19:59:16.534373 8980 log.cc:927] Time spent T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260620 19:59:16.494073 8969 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
W20260620 19:59:16.587199 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.602846 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 99098.6us
W20260620 19:59:16.631794 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 159781us
I20260620 19:59:16.569835 6905 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:16.579077 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:16.638907 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 74410.5us
I20260620 19:59:16.639317 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1969) took 1068 ms. Trace:
I20260620 19:59:16.647461 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 659) took 1138 ms. Trace:
W20260620 19:59:16.536670 8982 log.cc:927] Time spent T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20260620 19:59:16.603401 8643 consensus_meta.cc:287] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.734s user 0.008s sys 0.003s
W20260620 19:59:16.610692 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 116681us
W20260620 19:59:16.597350 8994 log.cc:927] Time spent T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260620 19:59:16.648023 6304 rpcz_store.cc:276] 0620 19:59:15.500689 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:15.508252 (+ 7563us) service_pool.cc:224] Handling call
0620 19:59:15.508583 (+ 331us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:15.844704 (+336121us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:16.621664 (+776960us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b54001cf554. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:16.633957 (+ 12293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":760085248}
W20260620 19:59:16.596536 8988 log.cc:927] Time spent T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260620 19:59:16.640301 6377 rpcz_store.cc:276] 0620 19:59:15.547762 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:15.564041 (+ 16279us) service_pool.cc:224] Handling call
0620 19:59:15.571574 (+ 7533us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:15.815000 (+243426us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:15.815046 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:15.839593 (+ 24547us) log.cc:844] Serialized 26 byte log entry
0620 19:59:16.008405 (+168812us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:16.008439 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:16.008508 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:16.615847 (+607339us) raft_consensus.cc:1685] finished
0620 19:59:16.615872 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:16.616009 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6785,"raft.queue_time_us":87201,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":389077,"threads_started":2,"wal-append.queue_time_us":341154}
W20260620 19:59:16.570070 8991 log.cc:927] Time spent T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.165s user 0.000s sys 0.004s
W20260620 19:59:16.600268 8798 consensus_meta.cc:287] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.627s user 0.008s sys 0.000s
I20260620 19:59:16.638653 8776 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:16.576651 8730 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:16.588591 9001 log.cc:927] Time spent T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260620 19:59:16.655171 6271 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:16.600944 8937 consensus_meta.cc:287] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.813s user 0.010s sys 0.003s
W20260620 19:59:16.596628 8989 log.cc:927] Time spent T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260620 19:59:16.588382 8983 log.cc:927] Time spent T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20260620 19:59:16.568867 6902 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:16.517397 8990 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.633741 8934 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:16.685422 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.569525 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:16.604457 8954 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:16.517946 8808 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:16.520282 8985 consensus_queue.cc:1059] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.210s
I20260620 19:59:16.535866 8976 consensus_queue.cc:237] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:16.827181 6874 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:16.704771 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:16.866988 6907 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:16.707281 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.798650 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 89521.1us
W20260620 19:59:16.799115 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 50652.2us
W20260620 19:59:16.870339 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.872685 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.873991 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.706966 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.704819 8829 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:16.723690 6344 leader_election.cc:336] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.553s (SENT)
W20260620 19:59:16.911790 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 241373us
W20260620 19:59:16.915556 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.916309 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:16.808473 6196 leader_election.cc:336] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325): Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 2.830s (SENT)
W20260620 19:59:16.822091 9003 log.cc:927] Time spent T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.308s user 0.000s sys 0.003s
W20260620 19:59:16.823490 9000 log.cc:927] Time spent T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260620 19:59:16.747898 8935 consensus_meta.cc:287] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.744s user 0.011s sys 0.000s
W20260620 19:59:16.763839 8981 log.cc:927] Time spent T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.685s user 0.012s sys 0.005s
I20260620 19:59:16.921998 8779 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:16.922401 8962 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:16.867305 8827 consensus_meta.cc:287] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.008s user 0.011s sys 0.000s
W20260620 19:59:16.820001 8957 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
W20260620 19:59:16.795435 8982 log.cc:927] Time spent T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260620 19:59:16.839323 8995 log.cc:927] Time spent T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.434s user 0.000s sys 0.004s
I20260620 19:59:16.920562 6904 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:16.920936 6377 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260620 19:59:16.917722 8730 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.723093 8643 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:16.925012 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 97893us
I20260620 19:59:16.820161 9005 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.913820 8934 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:16.982942 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 57317.1us
I20260620 19:59:16.692695 8618 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:16.918506 8993 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:17.008857 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:16.893869 8937 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.893272 8798 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:17.015141 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 206660us
I20260620 19:59:17.042565 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1594) took 1404 ms. Trace:
W20260620 19:59:17.021536 6877 leader_election.cc:336] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.541s (SENT)
I20260620 19:59:16.819792 8960 consensus_queue.cc:1059] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.264s
I20260620 19:59:17.050088 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:17.043040 6303 rpcz_store.cc:276] 0620 19:59:15.638177 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:15.898082 (+259905us) service_pool.cc:224] Handling call
0620 19:59:15.898476 (+ 394us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:15.921677 (+ 23201us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:15.921724 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:15.934541 (+ 12817us) log.cc:844] Serialized 26 byte log entry
0620 19:59:16.220118 (+285577us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:16.220155 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:16.220224 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:17.042350 (+822126us) raft_consensus.cc:1685] finished
0620 19:59:17.042373 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:17.042509 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23098,"raft.queue_time_us":62524,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":1318400,"thread_start_us":273484,"threads_started":1,"wal-append.queue_time_us":451520}
I20260620 19:59:17.050746 6902 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:17.049976 6906 raft_consensus.cc:1240] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:17.051510 8993 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.052402 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:17.053511 6907 raft_consensus.cc:2468] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:17.053829 8993 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:17.050662 8618 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.918843 8829 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:17.050705 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:17.060436 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.043236 6877 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:17.055436 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 124307us
I20260620 19:59:17.054451 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:17.020365 9000 log.cc:927] Time spent T 5ace7015fc514a909c3139c9e717b387 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20260620 19:59:17.017560 8865 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.013s sys 0.004s
W20260620 19:59:17.061519 6196 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.132:42325 timed out after 1.560s (SENT) [suppressed 47 similar messages]
I20260620 19:59:17.054834 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:16.955505 8827 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.010736 8935 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260620 19:59:17.084373 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:17.067752 8751 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:17.071956 8865 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.014s sys 0.004s
W20260620 19:59:17.086447 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 431293us
I20260620 19:59:17.084470 6910 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
W20260620 19:59:17.078195 6299 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.006s sys 0.043s
W20260620 19:59:17.105124 6299 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.006s sys 0.044s
I20260620 19:59:17.014426 8730 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:16.724385 8975 consensus_queue.cc:237] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:17.094319 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:17.120659 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 78779.6us
I20260620 19:59:17.122706 8944 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election won for term 5
W20260620 19:59:17.124159 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 102639us
I20260620 19:59:17.092074 6906 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:16.984375 8962 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.081820 8813 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:17.128453 8827 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:17.094376 8969 raft_consensus.cc:3055] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:17.130762 8730 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:17.131474 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:17.123709 6909 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
W20260620 19:59:17.134186 6877 leader_election.cc:336] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.717s (SENT)
I20260620 19:59:17.103499 6907 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:17.140038 6877 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:17.139935 6910 raft_consensus.cc:2376] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7. Re-sending same reply.
I20260620 19:59:17.144193 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:17.145820 6907 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:17.131603 8798 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:17.131714 8937 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:17.020102 9005 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.108368 8751 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:17.136058 8944 raft_consensus.cc:697] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:59:17.179682 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.155212 8999 consensus_queue.cc:1059] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260620 19:59:17.092341 9010 consensus_queue.cc:1059] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.063s
I20260620 19:59:17.141523 8969 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:17.195299 8643 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:17.144330 8985 consensus_queue.cc:1048] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260620 19:59:17.256673 6307 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260620 19:59:17.195564 8999 raft_consensus.cc:3055] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
W20260620 19:59:17.268112 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.211400 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1603) took 1563 ms. Trace:
W20260620 19:59:17.300741 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 244358us
W20260620 19:59:17.298719 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.153097 8814 consensus_queue.cc:1048] T 8b96addaf2b149a2ba7decbb292167e5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.251s
I20260620 19:59:17.196789 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:17.224531 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 79145.8us
W20260620 19:59:17.393244 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 237852us
I20260620 19:59:17.240267 6907 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:17.412595 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:17.214095 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 79918.2us
W20260620 19:59:17.393105 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 197012us
I20260620 19:59:17.211890 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:17.416355 6345 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:17.300468 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 238913us
I20260620 19:59:17.400370 6273 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:17.424537 8918 raft_consensus.cc:2804] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:17.416304 6909 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:17.425102 8918 raft_consensus.cc:697] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:17.424341 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:17.267966 8864 consensus_meta.cc:287] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.046s user 0.006s sys 0.005s
W20260620 19:59:17.311784 9022 log.cc:927] Time spent T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
W20260620 19:59:17.275139 8882 consensus_meta.cc:287] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.756s user 0.007s sys 0.003s
I20260620 19:59:17.384977 8808 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:17.432346 6877 leader_election.cc:336] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.742s (SENT)
I20260620 19:59:17.432593 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:17.393965 8990 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:17.392362 6902 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:17.404244 8829 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:17.275276 8934 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:17.407752 8999 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:17.398950 6300 rpcz_store.cc:276] 0620 19:59:15.647511 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:16.407710 (+760199us) service_pool.cc:224] Handling call
0620 19:59:16.443664 (+ 35954us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:16.470719 (+ 27055us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:16.470760 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:16.649090 (+178330us) log.cc:844] Serialized 26 byte log entry
0620 19:59:17.077144 (+428054us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:17.077183 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:17.077245 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:17.211175 (+133930us) raft_consensus.cc:1685] finished
0620 19:59:17.211193 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:17.211319 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":152,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":26507,"thread_start_us":322578,"threads_started":1,"wal-append.queue_time_us":471030}
I20260620 19:59:17.269014 8618 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:17.393477 8751 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:17.210884 8920 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:17.461390 6197 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.009s sys 0.034s
W20260620 19:59:17.461738 6197 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.544s user 0.010s sys 0.035s
I20260620 19:59:17.263113 8944 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.465626 6271 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:17.466918 8962 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:17.467060 6271 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:17.467208 8999 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.467880 8928 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:17.468560 8999 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:17.469092 6198 leader_election.cc:400] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Vote denied by peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701) with higher term. Message: Invalid argument: T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7.
I20260620 19:59:17.469271 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:17.470472 6198 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:17.470821 8814 raft_consensus.cc:2764] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260620 19:59:17.471519 6877 leader_election.cc:336] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.018s (SENT)
W20260620 19:59:17.472419 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.472026 8918 consensus_queue.cc:237] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.472079 6877 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:17.476212 6877 leader_election.cc:336] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.024s (SENT)
I20260620 19:59:17.476624 6877 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:17.404456 9010 raft_consensus.cc:3055] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:17.427649 8781 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:17.433804 8827 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:17.488003 9010 raft_consensus.cc:740] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:17.477531 8776 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Advancing to term 10
W20260620 19:59:17.487972 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 570927us
W20260620 19:59:17.513785 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.430503 6904 raft_consensus.cc:2468] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:17.514580 6345 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:17.515319 6271 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:17.471580 8814 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:17.526226 6910 raft_consensus.cc:2410] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:17.477871 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:17.489226 8990 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:17.535168 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 58987.6us
I20260620 19:59:17.517122 8993 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:17.513305 8730 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:17.516983 8882 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:17.545480 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.516170 8618 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260620 19:59:17.488781 8827 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:17.516350 8864 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260620 19:59:17.460279 8787 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.071s
W20260620 19:59:17.569432 6877 leader_election.cc:336] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.060s (SENT)
W20260620 19:59:17.575124 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 120676us
I20260620 19:59:17.546344 8781 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:17.580439 6345 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:17.581332 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 44006.1us
I20260620 19:59:17.547673 6906 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:17.608564 8934 raft_consensus.cc:2764] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
W20260620 19:59:17.600620 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 46628.2us
W20260620 19:59:17.612324 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:17.612663 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 58197.5us
W20260620 19:59:17.616631 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 93992.1us
I20260620 19:59:17.616967 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:17.554279 8808 consensus_queue.cc:1059] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260620 19:59:17.583545 8814 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:17.621954 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 143435us
I20260620 19:59:17.562759 8882 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:17.623472 6300 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260620 19:59:17.623675 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:17.629295 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:17.629472 6301 raft_consensus.cc:3055] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:17.611989 8985 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:17.630103 6301 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:17.632743 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 70163.9us
I20260620 19:59:17.580273 8993 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:17.634109 6306 raft_consensus.cc:2410] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:17.546185 8969 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:17.644788 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 91315.7us
I20260620 19:59:17.646664 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:17.647226 6300 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:17.599714 9029 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260620 19:59:17.650180 6301 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:17.649581 9015 log.cc:927] Time spent T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260620 19:59:17.657229 6345 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:17.660635 9028 log.cc:927] Time spent T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20260620 19:59:17.661417 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:17.662343 9029 raft_consensus.cc:3055] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:17.662791 9029 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:17.663600 9029 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.664845 9029 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:17.580232 9010 consensus_queue.cc:260] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.667265 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:17.667728 9010 raft_consensus.cc:3060] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:17.667845 6302 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:17.671242 6378 raft_consensus.cc:1275] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:17.672046 8993 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:17.674836 9021 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260620 19:59:17.675760 6302 raft_consensus.cc:2410] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:17.677030 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1593) took 2039 ms. Trace:
I20260620 19:59:17.677448 6299 rpcz_store.cc:276] 0620 19:59:15.637866 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:15.824060 (+186194us) service_pool.cc:224] Handling call
0620 19:59:15.843581 (+ 19521us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:16.444418 (+600837us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:16.444462 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:16.508229 (+ 63767us) log.cc:844] Serialized 26 byte log entry
0620 19:59:17.105619 (+597390us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:17.105672 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:17.105757 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:17.676804 (+571047us) raft_consensus.cc:1685] finished
0620 19:59:17.676829 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:17.676970 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100154,"raft.queue_time_us":114483,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":26796,"thread_start_us":1159651,"threads_started":2,"wal-append.queue_time_us":761348}
I20260620 19:59:17.680230 8976 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:17.686017 8776 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:17.687752 8882 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:17.689316 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 44737.7us
I20260620 19:59:17.666632 8730 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260620 19:59:17.695453 6304 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:17.697137 6301 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:17.706575 6300 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:17.719154 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:17.696389 8969 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:17.639647 8985 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:17.649448 8864 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260620 19:59:17.726042 6301 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260620 19:59:17.747012 6877 leader_election.cc:336] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.012s (SENT)
W20260620 19:59:17.734695 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 44676.7us
I20260620 19:59:17.659822 8808 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260620 19:59:17.720760 8962 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260620 19:59:17.775113 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 60857.9us
I20260620 19:59:17.713485 8993 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:17.777355 6271 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:17.766459 6877 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:17.779452 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 191552us
I20260620 19:59:17.780272 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:17.713253 8934 consensus_queue.cc:1059] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260620 19:59:17.813511 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:17.824083 6307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1597) took 2184 ms. Trace:
I20260620 19:59:17.831200 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:17.993659 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 217205us
W20260620 19:59:17.993261 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 215964us
I20260620 19:59:17.996610 6907 raft_consensus.cc:2410] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260620 19:59:17.998265 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:18.000128 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 219880us
I20260620 19:59:17.786661 8993 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:17.828714 9012 log.cc:927] Time spent T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.697s user 0.002s sys 0.000s
I20260620 19:59:17.771080 8882 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:17.778898 8781 raft_consensus.cc:2764] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260620 19:59:17.814769 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.026105 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.009598 6196 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:17.800182 8926 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:17.780699 9040 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:17.771847 8962 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:17.817267 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 707) took 1017 ms. Trace:
I20260620 19:59:17.995720 6910 raft_consensus.cc:2376] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6. Re-sending same reply.
W20260620 19:59:18.004863 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 224731us
I20260620 19:59:17.780830 8618 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:17.995373 6307 rpcz_store.cc:276] 0620 19:59:15.639291 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:16.112613 (+473322us) service_pool.cc:224] Handling call
0620 19:59:16.131248 (+ 18635us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:17.404467 (+1273219us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:17.823760 (+419293us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001a9354. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:17.824009 (+ 249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2587009408}
I20260620 19:59:17.815027 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:18.008882 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:18.010461 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 263457us
I20260620 19:59:18.035198 6907 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:18.035645 6877 leader_election.cc:336] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.196s (SENT)
I20260620 19:59:18.036190 6877 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:17.821535 9043 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:18.019042 6271 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:17.813156 6902 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260620 19:59:17.961767 8643 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:18.026829 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:17.815500 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:18.028224 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.041877 6228 raft_consensus.cc:2410] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:17.807597 8730 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:18.006397 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.037683 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:18.043848 6907 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 10.
I20260620 19:59:18.043238 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.043851 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.043668 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:18.050611 6908 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:18.044900 6231 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:18.029300 6911 rpcz_store.cc:276] 0620 19:59:16.799874 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:16.843036 (+ 43162us) service_pool.cc:224] Handling call
0620 19:59:16.843422 (+ 386us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:17.314173 (+470751us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:17.314220 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:17.450924 (+136704us) log.cc:844] Serialized 26 byte log entry
0620 19:59:17.565538 (+114614us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:17.565591 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:17.816814 (+251223us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54004b8054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:17.816862 (+ 48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:17.816905 (+ 43us) raft_consensus.cc:1685] finished
0620 19:59:17.816919 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:17.817115 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":147917,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":7243,"spinlock_wait_cycles":892012928,"thread_start_us":102321,"threads_started":1,"wal-append.queue_time_us":157790}
I20260620 19:59:18.047816 6905 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:18.047680 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:17.811707 6906 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:46646
I20260620 19:59:18.034013 8962 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:18.048851 6229 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:18.033492 8809 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:18.091593 8809 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:18.093560 8809 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:18.028854 8926 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.088182 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.087266 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:18.090168 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:18.095250 6907 raft_consensus.cc:2376] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6. Re-sending same reply.
I20260620 19:59:18.095932 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.096515 8993 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:18.096683 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 63500.8us
I20260620 19:59:18.098483 9055 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:18.099186 6345 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:18.028090 8882 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:18.101333 6345 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:18.101765 8882 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:18.102672 8934 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:18.102712 8730 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.103678 8934 raft_consensus.cc:2764] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260620 19:59:18.104043 6906 raft_consensus.cc:606] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.104447 6906 raft_consensus.cc:614] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:18.105039 8730 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:18.096223 8926 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:18.090122 6196 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.009469 8808 consensus_queue.cc:1059] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260620 19:59:18.093612 6874 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:18.108379 6906 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9"
dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
mode: GRACEFUL
new_leader_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
from {username='slave'} at 127.0.0.1:46646
I20260620 19:59:18.108986 6906 raft_consensus.cc:606] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Received request to transfer leadership to TS 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.109511 9002 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:18.110059 9002 raft_consensus.cc:697] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:18.110260 8730 raft_consensus.cc:2804] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:18.110194 8643 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:18.111099 8643 consensus_queue.cc:1243] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260620 19:59:18.111039 9002 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:18.113471 9047 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20260620 19:59:18.116187 6196 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.044514 8999 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:18.124687 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.042317 8985 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:18.149971 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.160236 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 129679us
W20260620 19:59:18.179991 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.180595 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.182361 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:18.185128 6378 raft_consensus.cc:1275] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260620 19:59:18.115406 9045 log.cc:927] Time spent T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Append to log took a long time: real 0.325s user 0.001s sys 0.000s
W20260620 19:59:18.186136 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 59435.7us
I20260620 19:59:18.188923 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:18.185192 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 166210us
I20260620 19:59:18.183123 6374 raft_consensus.cc:2376] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5. Re-sending same reply.
I20260620 19:59:18.189077 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.110932 8730 raft_consensus.cc:697] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:18.191785 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 156152us
W20260620 19:59:18.193706 6877 leader_election.cc:336] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.428s (SENT)
I20260620 19:59:18.194203 6877 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:18.197067 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:18.197434 6196 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:18.124682 8901 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:18.158574 8808 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:18.221242 8808 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:18.221699 8808 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:18.222788 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.224124 8985 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:18.224401 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.224925 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.224637 8985 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:18.128146 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.226656 6304 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:59:18.227165 6304 raft_consensus.cc:606] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:18.227550 6304 raft_consensus.cc:614] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:18.122150 8618 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.214819 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:18.230574 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.202661 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.044322 9043 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:18.203423 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.206883 6196 leader_election.cc:304] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.209635 9055 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:18.211102 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.210744 8730 consensus_queue.cc:237] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:18.213330 6877 leader_election.cc:336] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.297s (SENT)
I20260620 19:59:18.213539 8829 raft_consensus.cc:3055] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:18.213678 8776 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:18.214398 9036 log.cc:927] Time spent T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.547s user 0.000s sys 0.001s
I20260620 19:59:18.116852 9051 consensus_queue.cc:1059] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.165s
I20260620 19:59:18.227098 6223 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:18.232524 6381 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
W20260620 19:59:18.218367 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.215494 6379 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260620 19:59:18.222250 8993 raft_consensus.cc:3055] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:18.233044 6380 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:18.234938 6372 raft_consensus.cc:3055] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:18.208714 8827 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:18.244302 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.248114 6373 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:18.241081 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.249142 6377 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
W20260620 19:59:18.240463 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 50439.9us
I20260620 19:59:18.236667 8829 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:18.247016 6372 raft_consensus.cc:740] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:18.266846 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:18.236069 9055 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.284976 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.285962 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.258630 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:18.238538 8618 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:18.297374 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 90534.8us
I20260620 19:59:18.242630 8865 raft_consensus.cc:2804] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:18.299118 6305 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:18.298461 6196 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:18.303440 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 90150.8us
W20260620 19:59:18.304225 6877 leader_election.cc:336] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.675s (SENT)
I20260620 19:59:18.304636 6877 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:18.246222 8993 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:18.301688 6271 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.246798 8985 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:18.306558 6381 raft_consensus.cc:3055] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:18.311112 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.324115 6381 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:18.292552 9055 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:18.328986 6372 consensus_queue.cc:260] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:18.331864 6196 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.335572 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.336700 6303 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260620 19:59:18.340848 6372 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:18.343241 6303 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:18.334097 9062 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:18.358668 8882 raft_consensus.cc:3055] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:18.359160 8882 raft_consensus.cc:740] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:18.359736 9058 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260620 19:59:18.360131 9058 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.360219 6196 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:18.360672 9058 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Advancing to term 11
I20260620 19:59:18.361380 6196 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.358072 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:18.362957 9040 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:18.357610 6231 raft_consensus.cc:1240] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:18.351272 6232 raft_consensus.cc:1275] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260620 19:59:18.364660 9057 log.cc:927] Time spent T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260620 19:59:18.364899 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.353199 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.354575 6875 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:18.365830 6230 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:18.367213 6875 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:18.368553 8864 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:18.368480 8827 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:18.357270 6874 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:18.368556 9062 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:18.369161 8864 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.368697 8920 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:18.360069 8882 consensus_queue.cc:260] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.369671 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:18.370149 9010 consensus_meta.cc:287] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.702s user 0.008s sys 0.003s
W20260620 19:59:18.370291 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 268990us
I20260620 19:59:18.375289 6379 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [4.3-7.4]
I20260620 19:59:18.375130 8993 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.356920 6302 raft_consensus.cc:1275] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260620 19:59:18.363003 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 66859.6us
I20260620 19:59:18.356185 6299 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260620 19:59:18.335258 8618 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:18.348697 9005 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:18.344560 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.355324 6300 pending_rounds.cc:85] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 4
I20260620 19:59:18.351998 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:18.398128 6372 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260620 19:59:18.422331 6372 pending_rounds.cc:85] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 4
I20260620 19:59:18.391500 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:18.350119 9029 consensus_meta.cc:287] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.685s user 0.007s sys 0.005s
W20260620 19:59:18.416476 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.348367 6223 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:18.417608 6304 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0"
dest_uuid: "39662d01be7a44f682d160227f457661"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:60942
I20260620 19:59:18.422565 6372 pending_rounds.cc:107] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260620 19:59:18.427455 6304 raft_consensus.cc:606] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:18.427660 6372 raft_consensus.cc:2889] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:18.427834 6304 raft_consensus.cc:614] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:18.348624 6381 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:18.345641 8865 raft_consensus.cc:697] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:18.424980 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 225339us
I20260620 19:59:18.426208 6875 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:18.387811 8864 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:18.422003 6300 pending_rounds.cc:107] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260620 19:59:18.431777 6300 raft_consensus.cc:2889] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:18.390369 8829 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.432587 6381 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:18.432677 8829 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:18.388417 9051 raft_consensus.cc:2804] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:18.383713 9062 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.439296 9062 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:18.439908 8827 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:18.390570 8920 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:18.440940 8920 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:18.412937 9002 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:18.442482 9002 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:18.443536 9002 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.446817 6271 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.450030 6227 raft_consensus.cc:2410] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260620 19:59:18.450928 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.452509 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.456588 6875 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:18.458068 6875 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:18.453096 6372 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:18.460088 6381 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260620 19:59:18.467582 6875 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:18.468668 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.471439 6381 raft_consensus.cc:2410] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260620 19:59:18.472215 6374 raft_consensus.cc:1240] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:18.475616 6307 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:18.386065 8962 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.416875 8992 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260620 19:59:18.485977 6379 raft_consensus.cc:1240] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:18.482501 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.490041 6273 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:18.447659 8643 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260620 19:59:18.482717 8882 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:18.475658 9025 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:18.506178 6228 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:18.476508 8944 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:18.479626 8901 raft_consensus.cc:2764] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260620 19:59:18.482576 8934 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:18.505748 8776 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:18.430467 9005 consensus_queue.cc:1048] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260620 19:59:18.509613 8993 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:18.490922 6273 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:18.430783 9010 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.051s
I20260620 19:59:18.505820 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260620 19:59:18.523768 6376 raft_consensus.cc:2393] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 8: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:18.525326 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 78615.3us
I20260620 19:59:18.531580 9051 raft_consensus.cc:697] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:18.548483 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.479501 9041 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.054s
I20260620 19:59:18.522683 8985 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:18.480625 8827 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
W20260620 19:59:18.583297 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 92424.1us
W20260620 19:59:18.570040 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.552670 9043 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:18.591142 8751 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:18.563014 8944 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:18.594285 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 144896us
I20260620 19:59:18.607738 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.582855 8985 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:18.611426 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.614204 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.612452 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:18.606881 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 217174us
I20260620 19:59:18.617547 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.617697 6299 raft_consensus.cc:1275] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260620 19:59:18.617630 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.654765 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:18.617821 8974 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260620 19:59:18.618191 6304 raft_consensus.cc:1275] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260620 19:59:18.656528 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:18.608078 9049 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.485s user 0.003s sys 0.001s
I20260620 19:59:18.618688 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.619637 9051 consensus_queue.cc:237] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.659428 6299 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:18.660642 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.621366 9055 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.626837 9043 consensus_queue.cc:1048] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260620 19:59:18.630195 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.666294 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.667035 6378 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
W20260620 19:59:18.631044 9061 log.cc:927] Time spent T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
W20260620 19:59:18.634224 8926 consensus_meta.cc:287] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.528s user 0.006s sys 0.001s
I20260620 19:59:18.635850 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.637540 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 678) took 1023 ms. Trace:
W20260620 19:59:18.641314 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.643110 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:18.643729 6344 leader_election.cc:336] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.996s (SENT)
W20260620 19:59:18.676033 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.643721 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 55504.5us
I20260620 19:59:18.647217 6232 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260620 19:59:18.647614 8962 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:18.597539 8827 consensus_queue.cc:1243] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260620 19:59:18.553009 8934 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:18.651477 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.675144 6305 rpcz_store.cc:276] 0620 19:59:17.613784 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:17.613933 (+ 149us) service_pool.cc:224] Handling call
0620 19:59:17.619048 (+ 5115us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:18.302483 (+683435us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:18.637240 (+334757us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b5400141754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:18.637459 (+ 219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1558581504}
W20260620 19:59:18.677758 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.651705 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:18.657864 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 190380us
I20260620 19:59:18.680958 6372 raft_consensus.cc:1275] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260620 19:59:18.663285 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 47744us
I20260620 19:59:18.614270 8985 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:18.601588 8751 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:18.681087 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.683640 6198 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.676760 6232 pending_rounds.cc:85] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 4
I20260620 19:59:18.700454 6232 pending_rounds.cc:107] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260620 19:59:18.626178 9058 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.693892 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:18.701340 6907 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:18.694314 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:18.702318 9058 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:18.694589 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:18.702944 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:18.696187 6273 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:18.703692 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:18.704315 6304 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:18.696780 8934 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:18.697329 6271 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:18.698352 8751 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:18.693470 6902 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260620 19:59:18.704643 9027 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:18.705751 6232 raft_consensus.cc:2889] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:18.721220 6232 pending_rounds.cc:107] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260620 19:59:18.721477 6232 raft_consensus.cc:2889] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260620 19:59:18.716929 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.729207 6196 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:18.702914 9058 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:18.732003 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 48456.6us
I20260620 19:59:18.736449 6911 raft_consensus.cc:1240] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260620 19:59:18.601692 8865 consensus_queue.cc:237] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.739684 6198 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:18.704391 8999 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:18.744616 8787 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 19:59:18.681612 8926 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.745208 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.753506 6372 raft_consensus.cc:2393] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:18.745312 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:18.745929 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:18.731822 8808 consensus_meta.cc:287] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.505s user 0.009s sys 0.001s
I20260620 19:59:18.755357 6227 raft_consensus.cc:2410] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:18.746342 6223 raft_consensus.cc:1240] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260620 19:59:18.746289 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:18.747251 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:18.747740 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.759181 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:18.759895 6910 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Advancing to term 11
I20260620 19:59:18.759684 6196 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:18.669094 8864 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.749112 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 710) took 1885 ms. Trace:
I20260620 19:59:18.749574 8954 raft_consensus.cc:3055] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 LEADER]: Stepping down as leader of term 9
I20260620 19:59:18.749758 6378 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260620 19:59:18.755529 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 71949.3us
I20260620 19:59:18.754634 6226 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:18.761759 6198 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:18.747849 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:59:18.721060 9076 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:18.764575 9042 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:18.731228 9041 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:18.769731 8753 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260620 19:59:18.717783 8827 consensus_queue.cc:1048] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20260620 19:59:18.773052 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.743353 9046 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:18.776737 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.746321 9029 raft_consensus.cc:3055] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:18.792933 6379 raft_consensus.cc:2393] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:18.780881 6372 raft_consensus.cc:1240] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 2. Current term is 3. Ops: []
I20260620 19:59:18.781412 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.751386 8999 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:18.814877 6271 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.788911 6375 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:18.775156 9076 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.764806 8901 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260620 19:59:18.786209 8829 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:18.816659 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 55004.9us
I20260620 19:59:18.769855 9073 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:18.771695 8954 raft_consensus.cc:740] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:18.820174 6227 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:18.777438 9042 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.824267 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.751813 9043 consensus_queue.cc:1059] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260620 19:59:18.827929 6305 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260620 19:59:18.812139 9029 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:18.828838 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.828959 6377 raft_consensus.cc:1240] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:18.835225 6198 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.841854 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:18.767128 8808 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.820572 8999 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:18.815212 8787 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:18.774641 9077 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260620 19:59:18.832810 9073 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.843681 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:18.844624 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.848541 6230 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:18.850824 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:18.867413 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 89737.9us
I20260620 19:59:18.851703 6305 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:18.823767 9074 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:18.824186 9076 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:18.855389 6198 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:18.817600 8975 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:18.864650 6302 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260620 19:59:18.845093 9042 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:18.869110 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.790161 9041 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.886168 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:18.880323 6198 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.886914 6902 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:18.874902 6911 raft_consensus.cc:1240] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:18.884753 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:18.889600 6198 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:18.857808 9046 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:18.893784 6228 raft_consensus.cc:1240] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:18.839570 8809 consensus_queue.cc:1048] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260620 19:59:18.839794 9027 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260620 19:59:18.873723 9074 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:18.900905 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.902626 6372 raft_consensus.cc:2376] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7. Re-sending same reply.
I20260620 19:59:18.836066 8920 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.867070 8787 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.907544 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:18.908219 6911 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:18.897176 6229 raft_consensus.cc:1240] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:18.871274 9073 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:18.892127 9043 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:18.914422 6902 raft_consensus.cc:2410] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:18.891599 8827 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:18.924135 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:18.911675 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.866071 9058 consensus_queue.cc:1048] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260620 19:59:18.925323 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:18.925915 6902 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:18.858875 9067 consensus_queue.cc:1059] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260620 19:59:18.939145 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:18.915069 9074 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:18.919840 9027 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:18.895977 8618 consensus_queue.cc:1059] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260620 19:59:18.891330 8975 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.904078 9065 consensus_queue.cc:1059] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260620 19:59:18.914381 8901 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20260620 19:59:18.916983 8954 consensus_queue.cc:260] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:18.897557 9051 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20260620 19:59:19.332688 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.924454 8827 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:19.066161 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:19.376843 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.772795 6903 rpcz_store.cc:276] 0620 19:59:16.863972 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:16.953328 (+ 89356us) service_pool.cc:224] Handling call
0620 19:59:17.930223 (+976895us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800051998. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:17.947131 (+ 16908us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:18.006450 (+ 59319us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.008562 (+ 2112us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:18.105639 (+ 97077us) log.cc:844] Serialized 26 byte log entry
0620 19:59:18.299017 (+193378us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:18.307450 (+ 8433us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:18.328308 (+ 20858us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:18.748684 (+420376us) raft_consensus.cc:1685] finished
0620 19:59:18.748709 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:18.748917 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14322,"raft.queue_time_us":59249,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9738,"spinlock_wait_cycles":31819008,"thread_start_us":185096,"threads_started":1,"wal-append.queue_time_us":375415}
I20260620 19:59:19.059301 6902 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
W20260620 19:59:19.095064 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.370838 6231 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260620 19:59:19.377601 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.926312 9055 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:19.128093 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:19.013522 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:19.368160 6910 consensus_meta.cc:287] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.601s user 0.009s sys 0.012s
I20260620 19:59:19.384869 6910 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 11.
W20260620 19:59:19.362313 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:18.984342 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:19.264918 6907 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.009s sys 0.005s
W20260620 19:59:19.386927 6907 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.012s sys 0.006s
W20260620 19:59:19.352267 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 329585us
I20260620 19:59:19.387588 9077 raft_consensus.cc:3055] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:19.387993 9077 raft_consensus.cc:740] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:19.388723 9077 consensus_queue.cc:260] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:19.389559 9078 log.cc:927] Time spent T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.525s user 0.000s sys 0.002s
I20260620 19:59:19.391876 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:19.392692 6902 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:19.393494 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:19.121120 8882 consensus_meta.cc:287] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.573s user 0.008s sys 0.003s
I20260620 19:59:19.361393 6374 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
W20260620 19:59:19.274797 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:19.289836 9087 log.cc:927] Time spent T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
W20260620 19:59:19.361090 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 459453us
W20260620 19:59:19.104843 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.420917 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:19.005463 6223 raft_consensus.cc:1240] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 7. Current term is 8. Ops: []
W20260620 19:59:19.084837 9062 consensus_meta.cc:287] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.645s user 0.006s sys 0.004s
W20260620 19:59:19.005937 9081 log.cc:927] Time spent T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.172s user 0.000s sys 0.001s
I20260620 19:59:19.408787 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:19.409492 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.410465 6195 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:19.357460 9093 log.cc:927] Time spent T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260620 19:59:19.411283 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:19.410171 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:19.412336 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:19.390075 9077 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:19.412814 6911 raft_consensus.cc:2393] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 8: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:19.112834 8974 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:19.384290 6373 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:19.468961 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 420) took 1202 ms. Trace:
I20260620 19:59:19.493043 6225 rpcz_store.cc:276] 0620 19:59:18.231411 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.237382 (+ 5971us) service_pool.cc:224] Handling call
0620 19:59:19.433067 (+1195685us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b5400367ed4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:19.433227 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1899842304}
W20260620 19:59:19.418512 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:19.086862 8993 consensus_meta.cc:287] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.507s user 0.010s sys 0.000s
I20260620 19:59:18.987658 9005 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:19.285167 8985 consensus_meta.cc:287] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.564s user 0.006s sys 0.003s
I20260620 19:59:19.042865 8808 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:19.027936 8864 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:19.041455 8920 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:19.420405 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.420476 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:19.010855 8926 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:19.451229 8999 consensus_meta.cc:287] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.583s user 0.008s sys 0.001s
I20260620 19:59:19.426103 8954 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:19.380040 8827 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:19.461627 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 432) took 1077 ms. Trace:
I20260620 19:59:19.470346 6902 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:19.580405 6902 raft_consensus.cc:1240] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260620 19:59:19.462133 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49028 (request call id 1637) took 1241 ms. Trace:
I20260620 19:59:19.482193 6381 raft_consensus.cc:2376] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6. Re-sending same reply.
I20260620 19:59:19.407768 9010 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:19.420307 9073 consensus_meta.cc:287] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.501s user 0.000s sys 0.008s
I20260620 19:59:19.504663 8809 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:19.454838 8751 consensus_meta.cc:287] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.719s user 0.008s sys 0.002s
W20260620 19:59:19.461237 9042 consensus_meta.cc:287] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.564s user 0.005s sys 0.004s
I20260620 19:59:19.549463 8864 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:19.551616 6909 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
W20260620 19:59:19.556478 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:19.646963 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.557718 6230 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 11 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 9. Current term is 11. Ops: []
I20260620 19:59:19.452522 9062 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.581511 6375 rpcz_store.cc:276] 0620 19:59:18.188539 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.241576 (+ 53037us) service_pool.cc:224] Handling call
0620 19:59:19.429464 (+1187888us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b54000447d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:19.429626 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1136610688}
I20260620 19:59:19.579238 6232 rpcz_store.cc:276] 0620 19:59:18.352556 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.364970 (+ 12414us) service_pool.cc:224] Handling call
0620 19:59:18.646899 (+281929us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:18.721910 (+ 75011us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.430097 (+708187us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400078d50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:19.430260 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":647607552}
I20260620 19:59:19.649466 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:19.527370 8787 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:19.086555 9076 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:19.576994 6372 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.007s sys 0.014s
W20260620 19:59:19.586918 6344 leader_election.cc:336] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.703s (SENT)
W20260620 19:59:19.653398 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 66523.8us
I20260620 19:59:19.639038 9082 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:19.639914 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 750235us
W20260620 19:59:19.643630 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 232948us
I20260620 19:59:19.642307 6271 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:19.578751 6903 raft_consensus.cc:1240] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:19.675298 6902 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:19.676628 6910 raft_consensus.cc:1240] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:19.652510 6911 raft_consensus.cc:1240] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
W20260620 19:59:19.670271 6875 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.006s sys 0.031s
W20260620 19:59:19.677925 6875 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.604s user 0.006s sys 0.031s
W20260620 19:59:19.678424 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 604446us
I20260620 19:59:19.624567 8993 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:19.682940 6374 raft_consensus.cc:3055] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:19.683312 6374 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:59:19.680716 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.684636 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:19.686384 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 689) took 1333 ms. Trace:
W20260620 19:59:19.676836 6372 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s user 0.007s sys 0.014s
I20260620 19:59:19.635057 8926 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:19.682013 6196 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:19.689215 6196 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:19.690721 6875 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:19.690872 6907 raft_consensus.cc:1240] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:19.686892 6300 rpcz_store.cc:276] 0620 19:59:18.352683 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.354786 (+ 2103us) service_pool.cc:224] Handling call
0620 19:59:18.355124 (+ 338us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:18.448834 (+ 93710us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.448863 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:18.449361 (+ 498us) log.cc:844] Serialized 52 byte log entry
0620 19:59:18.610269 (+160908us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:18.610325 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:18.610411 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:19.686135 (+1075724us) raft_consensus.cc:1685] finished
0620 19:59:19.686162 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:19.686322 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38284,"raft.queue_time_us":18088,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":77,"thread_start_us":98786,"threads_started":1,"wal-append.queue_time_us":290841}
I20260620 19:59:19.696828 6227 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:19.697893 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1641) took 1330 ms. Trace:
I20260620 19:59:19.698318 8753 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:19.698822 8753 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:19.699303 8753 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:19.698288 6380 rpcz_store.cc:276] 0620 19:59:18.367287 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.376432 (+ 9145us) service_pool.cc:224] Handling call
0620 19:59:18.376761 (+ 329us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:18.394850 (+ 18089us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.394898 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:18.395490 (+ 592us) log.cc:844] Serialized 26 byte log entry
0620 19:59:18.677819 (+282329us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:18.677867 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:18.677929 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:19.388924 (+710995us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:19.682403 (+293479us) raft_consensus.cc:1685] finished
0620 19:59:19.682423 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:19.697827 (+ 15404us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10807,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":26814,"spinlock_wait_cycles":12959616,"thread_start_us":221701,"threads_started":1,"wal-append.queue_time_us":362035}
I20260620 19:59:19.691072 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:19.700796 6875 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:19.555090 8985 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.691659 6374 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:19.704667 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:19.704993 6374 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:19.705605 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 121058us
I20260620 19:59:19.706122 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:19.585206 8999 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:19.553923 9005 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.709666 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:19.680989 8787 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:19.636299 9010 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.681870 9099 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:19.640385 9073 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.720479 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:19.675725 9065 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260620 19:59:19.656299 8751 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:19.771973 8751 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.772336 6304 raft_consensus.cc:1240] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:19.773814 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1623) took 1150 ms. Trace:
I20260620 19:59:19.713465 8864 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:19.775135 8864 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:19.775657 9046 raft_consensus.cc:3055] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:19.776163 9046 raft_consensus.cc:740] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:19.777390 8864 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:19.777184 9046 consensus_queue.cc:260] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:19.778700 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.779865 8999 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.780122 9073 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:19.780633 9041 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:19.676529 9082 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.702006 8901 consensus_queue.cc:1059] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.055s
I20260620 19:59:19.774253 6304 rpcz_store.cc:276] 0620 19:59:18.623150 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.705947 (+ 82797us) service_pool.cc:224] Handling call
0620 19:59:18.706182 (+ 235us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:19.772881 (+1066699us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.773522 (+ 641us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003241d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:19.773744 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2450974592}
I20260620 19:59:19.711041 9043 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:19.742246 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:19.678185 8618 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20260620 19:59:19.750939 6902 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:19.753208 6305 raft_consensus.cc:1240] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:19.642846 8809 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.671430 9042 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.759176 6271 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:19.759758 9076 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.760231 6911 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:19.761787 9010 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.657406 9029 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:19.768699 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:19.792248 6374 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260620 19:59:19.716223 8808 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:19.706638 8920 consensus_queue.cc:1048] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
I20260620 19:59:19.778451 9046 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:19.808350 9042 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:19.778723 9098 log.cc:927] Time spent T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20260620 19:59:19.787247 9082 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.755817 9055 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:19.721249 9051 raft_consensus.cc:2804] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 11 FOLLOWER]: Leader election won for term 11
I20260620 19:59:19.707684 8926 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.690734 8993 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.825256 6374 raft_consensus.cc:3055] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:19.825908 6304 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:19.842971 6374 raft_consensus.cc:740] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:19.804380 9043 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:19.821276 8808 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:19.810395 9094 log.cc:927] Time spent T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260620 19:59:19.788089 9099 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:19.824884 8751 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:19.809834 9010 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260620 19:59:19.847813 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 695) took 1176 ms. Trace:
I20260620 19:59:19.831547 8975 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.830438 9082 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:19.869215 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 169312us
I20260620 19:59:19.862879 6303 rpcz_store.cc:276] 0620 19:59:18.671618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.820679 (+149061us) service_pool.cc:224] Handling call
0620 19:59:18.820965 (+ 286us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:18.827749 (+ 6784us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.827777 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:18.851853 (+ 24076us) log.cc:844] Serialized 26 byte log entry
0620 19:59:19.103631 (+251778us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:19.103689 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.103879 (+ 190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:19.847567 (+743688us) raft_consensus.cc:1685] finished
0620 19:59:19.847592 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:19.847744 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6381,"raft.queue_time_us":15802,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7069,"thread_start_us":192169,"threads_started":1,"wal-append.queue_time_us":612440}
I20260620 19:59:19.835335 9051 raft_consensus.cc:697] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [term 11 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:19.876508 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 160855us
W20260620 19:59:19.881102 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.878727 6374 consensus_queue.cc:260] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:19.886456 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:19.867367 9105 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260620 19:59:19.846374 9062 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.830519 9055 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:19.914754 6374 raft_consensus.cc:3060] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:19.916833 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 227644us
I20260620 19:59:19.880980 9042 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:19.924443 6877 connection.cc:256] RPC call timeout handler was delayed by -1.3219346570000425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20260620 19:59:19.943147 6196 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:19.943336 6877 leader_election.cc:336] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.718s (SENT)
W20260620 19:59:19.944406 6877 leader_election.cc:336] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.845s (SENT)
I20260620 19:59:19.944891 6877 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:19.946213 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.901340 8999 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:19.949915 6271 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:19.913821 8985 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:19.953954 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.926961 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 860) took 1005 ms. Trace:
W20260620 19:59:19.954438 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:19.958220 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.959362 6908 raft_consensus.cc:1240] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 9. Current term is 11. Ops: []
W20260620 19:59:19.953794 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 253021us
I20260620 19:59:19.956043 6198 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:19.968948 6374 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260620 19:59:19.978490 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:19.980073 6374 pending_rounds.cc:85] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 6
I20260620 19:59:19.980960 6374 pending_rounds.cc:107] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260620 19:59:19.981750 6374 raft_consensus.cc:2889] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:19.985208 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 857) took 1079 ms. Trace:
I20260620 19:59:19.986696 6230 raft_consensus.cc:2393] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 8: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:19.988022 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:19.958771 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:19.952425 9062 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:19.985733 6908 rpcz_store.cc:276] 0620 19:59:18.883582 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.958933 (+1075351us) service_pool.cc:224] Handling call
0620 19:59:19.959201 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:19.959745 (+ 544us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.962523 (+ 2778us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:19.991101 6232 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260620 19:59:19.995254 6232 pending_rounds.cc:85] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 6
I20260620 19:59:19.995486 6232 pending_rounds.cc:107] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260620 19:59:19.995769 6232 raft_consensus.cc:2889] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:19.933756 8920 consensus_queue.cc:1059] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260620 19:59:19.953064 9065 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:19.955502 9101 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
W20260620 19:59:20.041828 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 85862.6us
W20260620 19:59:20.042394 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:20.043439 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 99047.3us
I20260620 19:59:19.978379 8969 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:20.043489 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 110623us
I20260620 19:59:19.955986 8829 raft_consensus.cc:3055] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:19.951499 9042 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:20.046365 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 56021.9us
I20260620 19:59:20.057313 6908 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:20.057673 6907 raft_consensus.cc:1275] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:20.064483 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:20.065084 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:20.065577 9128 raft_consensus.cc:3055] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:20.065727 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.065997 9128 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:20.066840 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:20.066883 9128 consensus_queue.cc:260] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.068125 6378 raft_consensus.cc:2468] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 10.
I20260620 19:59:20.068284 9128 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:20.068683 9111 log.cc:927] Time spent T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20260620 19:59:20.064534 8827 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:20.069931 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 1994) took 1405 ms. Trace:
I20260620 19:59:20.070358 8808 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:20.071045 9085 log.cc:927] Time spent T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 1.212s user 0.003s sys 0.000s
I20260620 19:59:20.071839 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2096) took 1046 ms. Trace:
I20260620 19:59:20.067535 6304 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:20.072307 9100 log.cc:927] Time spent T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.238s user 0.000s sys 0.000s
I20260620 19:59:20.072188 6231 rpcz_store.cc:276] 0620 19:59:19.025544 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.025744 (+ 200us) service_pool.cc:224] Handling call
0620 19:59:19.370470 (+344726us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:19.379812 (+ 9342us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:20.071571 (+691759us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b540030afa0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88321a0 00007fa6b88bdde2 00007fa6b88ba1b6 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22
0620 19:59:20.071782 (+ 211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9520896}
I20260620 19:59:20.048471 9114 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:19.955621 9076 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.080s
I20260620 19:59:20.075661 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 698) took 1380 ms. Trace:
W20260620 19:59:20.042181 9108 log.cc:927] Time spent T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260620 19:59:20.083909 6911 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260620 19:59:20.035568 8901 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:20.055828 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 740) took 1374 ms. Trace:
I20260620 19:59:20.090059 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:20.087756 6198 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:20.042500 8809 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:20.046888 9010 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20260620 19:59:20.095252 8827 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:20.095784 9055 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:20.093513 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:20.084991 6299 rpcz_store.cc:276] 0620 19:59:18.694854 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.846290 (+151436us) service_pool.cc:224] Handling call
0620 19:59:18.864783 (+ 18493us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:18.865275 (+ 492us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.865316 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:18.875453 (+ 10137us) log.cc:844] Serialized 52 byte log entry
0620 19:59:19.140430 (+264977us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:19.140469 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.140536 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:20.075458 (+934922us) raft_consensus.cc:1685] finished
0620 19:59:20.075478 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:20.075606 (+ 128us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9047,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6337,"thread_start_us":226781,"threads_started":1,"wal-append.queue_time_us":927855}
I20260620 19:59:20.097867 8808 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260620 19:59:20.098071 9055 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:20.099332 6372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2001) took 1178 ms. Trace:
I20260620 19:59:20.097900 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.099709 6372 rpcz_store.cc:276] 0620 19:59:18.920963 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.931157 (+ 10194us) service_pool.cc:224] Handling call
0620 19:59:18.996013 (+ 64856us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:18.996440 (+ 427us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.996485 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:19.025638 (+ 29153us) log.cc:844] Serialized 26 byte log entry
0620 19:59:19.688501 (+662863us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:19.688541 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.688603 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:20.099136 (+410533us) raft_consensus.cc:1685] finished
0620 19:59:20.099156 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:20.099279 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":562,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":82,"thread_start_us":657881,"threads_started":1,"wal-append.queue_time_us":779254}
I20260620 19:59:20.100916 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:20.101066 8993 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:20.101390 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:20.101913 6372 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:20.102001 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.102952 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 739) took 1421 ms. Trace:
I20260620 19:59:20.101661 6304 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:20.103142 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.101728 9112 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:20.104228 6304 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:20.105196 8809 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:20.105623 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:20.020251 9002 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260620 19:59:20.020908 9062 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.106513 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:20.121248 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:20.124197 6907 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:19.958738 8999 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.094501 6198 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:20.128726 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.130744 6372 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:20.124516 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:20.052902 9005 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:20.120364 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:20.113976 9112 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:20.053614 9101 raft_consensus.cc:697] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:20.053874 8829 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:20.070282 6376 rpcz_store.cc:276] 0620 19:59:18.664878 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.665050 (+ 172us) service_pool.cc:224] Handling call
0620 19:59:18.680610 (+ 15560us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:18.680995 (+ 385us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.681021 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:18.746311 (+ 65290us) log.cc:844] Serialized 52 byte log entry
0620 19:59:19.455386 (+709075us) spinlock_profiling.cc:243] Waited 4.99 ms on lock 0x7b180044b2a0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b8459471 00007fa6b8875def
0620 19:59:19.792361 (+336975us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:19.792432 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.792627 (+ 195us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:20.069639 (+277012us) raft_consensus.cc:1685] finished
0620 19:59:20.069662 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:20.069876 (+ 214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10385,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":44493,"spinlock_wait_cycles":11483904,"thread_start_us":152290,"threads_started":1,"wal-append.queue_time_us":1113899}
I20260620 19:59:20.139884 6271 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:20.086813 6905 rpcz_store.cc:276] 0620 19:59:18.681501 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.682239 (+ 738us) service_pool.cc:224] Handling call
0620 19:59:18.682575 (+ 336us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:18.721286 (+ 38711us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.721317 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:18.760916 (+ 39599us) log.cc:844] Serialized 52 byte log entry
0620 19:59:18.928537 (+167621us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:18.928580 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:18.928636 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:19.084757 (+156121us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b4800060fd8. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6bd7e41a1 00007fa6bd7db237 00007fa6bd7db0ca 00007fa6bd8d296f 00007fa6bd8d2637 00007fa6bd8e6c5c 00007fa6bd8e6c08 00007fa6bd8e6b88 00007fa6bd8e6b43 00007fa6bd8e5dec 00007fa6bc812f76
0620 19:59:19.993190 (+908433us) raft_consensus.cc:1685] finished
0620 19:59:19.993219 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:20.055746 (+ 62527us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24844,"raft.queue_time_us":36762,"raft.run_cpu_time_us":2997,"raft.run_wall_time_us":358168,"spinlock_wait_cycles":12494208,"thread_start_us":109675,"threads_started":1,"wal-append.queue_time_us":800473}
W20260620 19:59:20.092181 9109 log.cc:927] Time spent T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
W20260620 19:59:20.107223 9074 consensus_meta.cc:287] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.698s user 0.010s sys 0.003s
I20260620 19:59:20.156165 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:20.158210 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:20.150970 8969 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:20.161402 6907 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:20.158185 6196 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:20.166587 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:19.957965 9051 consensus_queue.cc:237] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.172063 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:20.119899 9099 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:19.957448 9043 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.175462 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:20.176685 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:20.177340 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:20.178032 6376 raft_consensus.cc:2410] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260620 19:59:20.115600 8926 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:20.178323 6902 raft_consensus.cc:3060] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:20.119191 9042 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:20.137964 8809 raft_consensus.cc:2764] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260620 19:59:20.137720 9046 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:20.044539 9065 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.184499 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:20.185201 6228 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:20.187076 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:20.187254 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:20.187839 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:20.190019 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.112668 8808 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.066s
I20260620 19:59:20.134378 8618 consensus_queue.cc:1048] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260620 19:59:20.194159 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.191447 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:20.160293 9112 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.195236 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:20.186656 8730 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:20.195118 9101 consensus_queue.cc:237] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.195606 6230 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:20.166564 8985 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:20.196113 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 56291.3us
I20260620 19:59:20.232228 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:20.172286 9074 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:20.187161 9099 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:20.197777 6875 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:20.220369 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:20.236362 6374 raft_consensus.cc:2393] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:20.226099 6902 raft_consensus.cc:2468] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:20.227310 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:20.237293 6376 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:20.229971 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 54926.3us
I20260620 19:59:20.109680 8753 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.179525 9005 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260620 19:59:20.173060 9073 consensus_queue.cc:1059] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260620 19:59:20.244659 9112 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:20.245539 9099 raft_consensus.cc:697] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:20.103389 6904 rpcz_store.cc:276] 0620 19:59:18.681232 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.692978 (+ 11746us) service_pool.cc:224] Handling call
0620 19:59:18.693333 (+ 355us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:18.743746 (+ 50413us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.748463 (+ 4717us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:18.748993 (+ 530us) log.cc:844] Serialized 26 byte log entry
0620 19:59:19.832101 (+1083108us) spinlock_profiling.cc:243] Waited 7.07 ms on lock 0x7b1800454f00. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b8459471 00007fa6b8875def
0620 19:59:19.978909 (+146808us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:20.005339 (+ 26430us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:20.045443 (+ 40104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:20.102652 (+ 57209us) raft_consensus.cc:1685] finished
0620 19:59:20.102685 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:20.102889 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25059,"raft.queue_time_us":33100,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":10381,"spinlock_wait_cycles":16265984,"thread_start_us":123179,"threads_started":1,"wal-append.queue_time_us":1298900}
I20260620 19:59:20.246487 9099 consensus_queue.cc:237] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:20.239212 6909 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:20.248019 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:20.233354 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 53034.8us
I20260620 19:59:20.248631 6300 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 3.
I20260620 19:59:20.238479 6196 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:20.180948 8865 consensus_queue.cc:1059] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.052s
I20260620 19:59:20.178740 8969 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:20.248891 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 67222.1us
I20260620 19:59:20.227262 8730 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:20.254518 6229 raft_consensus.cc:2393] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:20.257251 6908 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:20.263773 6374 raft_consensus.cc:3055] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:20.264717 6374 raft_consensus.cc:740] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:20.236476 8751 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:20.236972 9010 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:20.269717 6374 consensus_queue.cc:260] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.231815 8827 consensus_queue.cc:1048] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260620 19:59:20.275166 6301 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1626) took 1448 ms. Trace:
I20260620 19:59:20.279701 6374 raft_consensus.cc:3060] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:20.283413 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.285455 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:20.286774 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.255326 9076 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:20.275630 6301 rpcz_store.cc:276] 0620 19:59:18.827289 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.852293 (+ 25004us) service_pool.cc:224] Handling call
0620 19:59:18.872965 (+ 20672us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:19.236846 (+363881us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:19.236890 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:19.372456 (+135566us) log.cc:844] Serialized 52 byte log entry
0620 19:59:19.714267 (+341811us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:19.714307 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.714369 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:20.274897 (+560528us) raft_consensus.cc:1685] finished
0620 19:59:20.274925 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:20.275097 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1897,"raft.queue_time_us":22921,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":7876,"thread_start_us":639214,"threads_started":2,"wal-append.queue_time_us":446277}
I20260620 19:59:20.289831 6195 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:20.267532 8730 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:20.293066 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:20.299556 6380 raft_consensus.cc:1240] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:20.300922 6380 raft_consensus.cc:3060] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:20.255702 9042 consensus_queue.cc:1059] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260620 19:59:20.308936 6301 raft_consensus.cc:1217] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.3->[6.4-6.4] Dedup: 6.4->[]
I20260620 19:59:20.257439 9005 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260620 19:59:20.309504 6301 raft_consensus.cc:1240] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260620 19:59:20.268204 9096 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:20.305083 6374 raft_consensus.cc:1275] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:20.288558 9135 raft_consensus.cc:3055] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:20.294380 9076 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:20.287840 9062 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:20.333941 6299 raft_consensus.cc:1240] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260620 19:59:20.302533 8865 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:20.344815 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2092) took 1505 ms. Trace:
I20260620 19:59:20.318994 8753 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:20.345171 6227 rpcz_store.cc:276] 0620 19:59:18.839948 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.846275 (+ 6327us) service_pool.cc:224] Handling call
0620 19:59:18.864078 (+ 17803us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:19.708258 (+844180us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:20.342482 (+634224us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b5400077954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:20.344756 (+ 2274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1914369920}
I20260620 19:59:20.354527 6299 raft_consensus.cc:1240] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260620 19:59:20.336454 9076 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:20.326720 9135 raft_consensus.cc:740] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:20.361903 6299 raft_consensus.cc:1217] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 5.4->[7.5-7.5] Dedup: 7.5->[]
I20260620 19:59:20.362324 6299 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260620 19:59:20.360255 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 94763.9us
I20260620 19:59:20.367681 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:19.957043 6906 rpcz_store.cc:276] 0620 19:59:18.910293 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.549366 (+639073us) service_pool.cc:224] Handling call
0620 19:59:19.549785 (+ 419us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:19.639972 (+ 90187us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:19.657684 (+ 17712us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:19.769506 (+111822us) log.cc:844] Serialized 52 byte log entry
0620 19:59:19.871261 (+101755us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:19.872291 (+ 1030us) log.cc:844] Serialized 14 byte log entry
0620 19:59:19.872550 (+ 259us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.872652 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:19.881879 (+ 9227us) raft_consensus.cc:1685] finished
0620 19:59:19.881902 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:19.915735 (+ 33833us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6908,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":125996,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":30614,"thread_start_us":101220,"threads_started":1,"wal-append.queue_time_us":101719}
I20260620 19:59:20.374199 6299 raft_consensus.cc:1217] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.6->[6.7-6.7] Dedup: 6.7->[]
I20260620 19:59:20.337430 9096 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:20.671326 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 856) took 1799 ms. Trace:
I20260620 19:59:20.437381 6376 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:20.466437 9005 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:20.503584 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:20.435343 6380 raft_consensus.cc:1275] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260620 19:59:20.658982 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:20.546960 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:20.435314 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:20.700443 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:20.472261 6905 raft_consensus.cc:1275] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:18.996676 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:59:20.429066 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:20.681330 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:20.697937 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:20.363632 8999 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:20.701606 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.021579 6228 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:20.997498 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:20.997568 6298 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.001s sys 0.010s
W20260620 19:59:20.829931 6378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 515) took 1954 ms (client timeout 2018 ms). Trace:
I20260620 19:59:21.016400 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.006050 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1664) took 1009 ms. Trace:
W20260620 19:59:20.466107 9130 log.cc:927] Time spent T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20260620 19:59:21.033365 6905 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:21.033232 6380 rpcz_store.cc:276] 0620 19:59:19.996918 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.300473 (+303555us) service_pool.cc:224] Handling call
0620 19:59:20.300788 (+ 315us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:21.005821 (+705033us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.005998 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:21.002477 6907 raft_consensus.cc:1240] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:21.001906 6299 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260620 19:59:20.379822 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:21.019371 6911 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:21.017674 6230 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
W20260620 19:59:21.025560 6877 connection.cc:256] RPC call timeout handler was delayed by -1.0577746089999209s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
W20260620 19:59:20.571190 9131 log.cc:927] Time spent T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20260620 19:59:20.566560 9114 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:20.670495 9128 consensus_meta.cc:287] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.602s user 0.009s sys 0.000s
I20260620 19:59:20.427141 9135 consensus_queue.cc:260] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.055594 9135 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:21.057972 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.058624 6902 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:20.424889 8829 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.059476 8829 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:21.059481 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:20.828193 9095 log.cc:927] Time spent T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 1.084s user 0.004s sys 0.000s
I20260620 19:59:21.061983 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 693) took 2407 ms. Trace:
I20260620 19:59:21.062315 6306 rpcz_store.cc:276] 0620 19:59:18.654772 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.758657 (+103885us) service_pool.cc:224] Handling call
0620 19:59:18.780173 (+ 21516us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:18.798351 (+ 18178us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:18.798397 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:18.798970 (+ 573us) log.cc:844] Serialized 26 byte log entry
0620 19:59:18.927402 (+128432us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:18.928280 (+ 878us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:18.928365 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:20.827768 (+1899403us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x7b240000db68. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b564f614 00007fa6b564d71b 00007fa6b5808619 00007fa6b58082a5 00007fa6b58078ed 00007fa6b843d4ae 00007fa6b8454c2c 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1
0620 19:59:21.061696 (+233928us) raft_consensus.cc:1685] finished
0620 19:59:21.061716 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.061912 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13868,"raft.queue_time_us":11181,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":22795,"spinlock_wait_cycles":12982528,"thread_start_us":69041,"threads_started":1,"wal-append.queue_time_us":826146}
I20260620 19:59:21.064267 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:21.065399 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.066372 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.067060 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.027595 9041 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.067816 6306 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:21.068501 9041 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:20.667554 9107 log.cc:927] Time spent T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.979s user 0.001s sys 0.001s
W20260620 19:59:20.661530 9084 log.cc:927] Time spent T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 1.377s user 0.004s sys 0.000s
I20260620 19:59:21.031025 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.034619 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:21.070744 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:21.032207 6378 rpcz_store.cc:269] 0620 19:59:18.875453 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.875638 (+ 185us) service_pool.cc:224] Handling call
0620 19:59:20.829694 (+1954056us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400120cd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:20.829856 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2656526208}
W20260620 19:59:20.563483 9129 log.cc:927] Time spent T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20260620 19:59:21.040556 6227 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:20.701843 9025 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:20.704345 8975 raft_consensus.cc:3055] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:21.073974 9041 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.083578 8975 raft_consensus.cc:740] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:20.991926 8808 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:20.483358 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:20.403863 8969 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.034268 8999 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.046052 6224 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:21.087527 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45810 (request call id 2098) took 2005 ms. Trace:
I20260620 19:59:20.999114 6903 rpcz_store.cc:276] 0620 19:59:18.872327 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.450129 (+577802us) service_pool.cc:224] Handling call
0620 19:59:19.504474 (+ 54345us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:19.674966 (+170492us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:20.671124 (+996158us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x7b54002a3350. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:20.671265 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91181952}
W20260620 19:59:21.049962 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.088603 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:21.089179 6228 raft_consensus.cc:3060] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:21.054354 8730 consensus_meta.cc:287] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.745s user 0.009s sys 0.000s
I20260620 19:59:20.659351 8954 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:21.070818 6301 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260620 19:59:21.072556 6375 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [4.3-7.4]
I20260620 19:59:21.043454 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.094449 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.070093 6307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 701) took 2159 ms. Trace:
I20260620 19:59:21.074626 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.075381 6376 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:20.430550 6229 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45810 (request call id 2097) took 1354 ms (client timeout 1743 ms). Trace:
I20260620 19:59:21.041401 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:21.038841 6298 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s user 0.001s sys 0.010s
I20260620 19:59:21.090035 6196 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:21.087978 6224 rpcz_store.cc:276] 0620 19:59:19.081945 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.373108 (+291163us) service_pool.cc:224] Handling call
0620 19:59:19.408155 (+ 35047us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:21.045892 (+1637737us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b54003082a0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88321a0 00007fa6b88bc5a0 00007fa6b88bd611 00007fa6b88ba16f 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c
0620 19:59:21.087293 (+ 41401us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.087473 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2803968}
W20260620 19:59:21.097945 6229 rpcz_store.cc:269] 0620 19:59:19.075927 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.373352 (+297425us) service_pool.cc:224] Handling call
0620 19:59:20.430375 (+1057023us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b540004c4d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:20.430498 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1826431232}
I20260620 19:59:21.040113 9112 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20260620 19:59:21.095232 6307 rpcz_store.cc:276] 0620 19:59:18.911007 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.998474 (+ 87467us) service_pool.cc:224] Handling call
0620 19:59:19.025715 (+ 27241us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:19.047506 (+ 21791us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:19.047563 (+ 57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:19.094368 (+ 46805us) log.cc:844] Serialized 26 byte log entry
0620 19:59:19.369017 (+274649us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:19.369062 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:19.369138 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:20.666896 (+1297758us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:21.069845 (+402949us) raft_consensus.cc:1685] finished
0620 19:59:21.069865 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.070043 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6884,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":20134,"spinlock_wait_cycles":5031680,"thread_start_us":193135,"threads_started":1,"wal-append.queue_time_us":491569}
I20260620 19:59:21.107617 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.108246 6224 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
W20260620 19:59:21.078755 9062 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.009s sys 0.025s
W20260620 19:59:21.103987 6223 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.006s sys 0.022s
I20260620 19:59:21.111398 6196 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:21.112888 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.093641 9073 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.080219 9005 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:21.108779 8985 consensus_meta.cc:287] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.874s user 0.011s sys 0.000s
I20260620 19:59:21.090538 8808 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.119535 6304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1637) took 1029 ms. Trace:
I20260620 19:59:21.120215 6196 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:21.117204 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:21.116367 6223 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s user 0.006s sys 0.022s
W20260620 19:59:21.112468 9062 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.010s sys 0.025s
I20260620 19:59:21.131067 6905 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.131273 6228 raft_consensus.cc:2468] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:21.127166 6304 rpcz_store.cc:276] 0620 19:59:20.089958 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.105094 (+ 15136us) service_pool.cc:224] Handling call
0620 19:59:20.105391 (+ 297us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:20.105755 (+ 364us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:20.105783 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:20.106309 (+ 526us) log.cc:844] Serialized 52 byte log entry
0620 19:59:20.168682 (+ 62373us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:20.168736 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:20.168841 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.119281 (+950440us) raft_consensus.cc:1685] finished
0620 19:59:21.119304 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.119478 (+ 174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1532,"raft.queue_time_us":22185,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":20015,"thread_start_us":61952,"threads_started":1,"wal-append.queue_time_us":133094}
I20260620 19:59:21.115893 9114 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:21.141703 9114 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:21.104905 8954 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.102553 8730 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.088238 9025 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.093413 8999 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.116952 8864 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260620 19:59:21.137718 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.135224 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.150624 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 706) took 1086 ms. Trace:
I20260620 19:59:21.160912 6303 rpcz_store.cc:276] 0620 19:59:20.063821 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.066697 (+ 2876us) service_pool.cc:224] Handling call
0620 19:59:20.067010 (+ 313us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:20.067394 (+ 384us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:20.067423 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:20.067873 (+ 450us) log.cc:844] Serialized 26 byte log entry
0620 19:59:20.269232 (+201359us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:20.269303 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:20.269396 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.150307 (+880911us) raft_consensus.cc:1685] finished
0620 19:59:21.150397 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.150534 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2022,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":82,"thread_start_us":200978,"threads_started":1,"wal-append.queue_time_us":266248}
I20260620 19:59:21.125931 8985 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:21.091085 8975 consensus_queue.cc:260] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.152403 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:21.125418 9082 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.142095 9114 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.123838 9073 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.149806 9115 raft_consensus.cc:2804] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:21.170120 9115 raft_consensus.cc:697] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:21.171383 9115 consensus_queue.cc:237] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.189465 6306 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:21.194383 6225 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:21.204339 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:21.204124 6229 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:21.189092 8975 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:21.236975 8901 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.237596 8901 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.239629 8901 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:21.245513 6905 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:21.247478 6226 raft_consensus.cc:2410] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:21.248478 6223 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 441) took 1079 ms. Trace:
I20260620 19:59:21.249647 8999 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.204668 9074 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:21.250125 9154 log.cc:927] Time spent T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20260620 19:59:21.169315 9082 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.251713 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 694) took 2596 ms. Trace:
I20260620 19:59:21.252115 9082 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:21.252192 6298 rpcz_store.cc:276] 0620 19:59:18.655734 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.765295 (+109561us) service_pool.cc:224] Handling call
0620 19:59:18.834175 (+ 68880us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:20.282015 (+1447840us) spinlock_profiling.cc:243] Waited 9.74 ms on lock 0x7b180044ab80. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c5c55 00007fa6b88b2440 00007fa6b88b2fdb 00007fa6b88bda2b 00007fa6b88ba16f
0620 19:59:20.350907 (+ 68892us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:20.350956 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:20.406689 (+ 55733us) log.cc:844] Serialized 26 byte log entry
0620 19:59:21.109809 (+703120us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:21.113635 (+ 3826us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.117863 (+ 4228us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.251334 (+133471us) raft_consensus.cc:1685] finished
0620 19:59:21.251366 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.251633 (+ 267us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6928,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":22398848,"thread_start_us":751311,"threads_started":2,"wal-append.queue_time_us":677464}
I20260620 19:59:21.254884 9005 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:21.255290 9152 log.cc:927] Time spent T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260620 19:59:21.257882 9073 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:21.259555 6229 raft_consensus.cc:1240] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: []
I20260620 19:59:21.260915 6875 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:21.261967 6875 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:21.203254 9156 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:21.278909 6375 raft_consensus.cc:3060] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Advancing to term 11
I20260620 19:59:21.215996 9096 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.317077 8808 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:21.287154 9164 log.cc:927] Time spent T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260620 19:59:21.267290 9074 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.326270 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.328097 6376 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:21.288733 9005 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:21.339640 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.343011 6378 raft_consensus.cc:2410] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260620 19:59:21.287431 9082 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260620 19:59:21.343977 9124 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.016s sys 0.048s
I20260620 19:59:21.347280 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.348014 6229 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
W20260620 19:59:21.347361 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 311599us
I20260620 19:59:21.323678 8901 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.341784 9005 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:21.365607 6877 leader_election.cc:336] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.693s (SENT)
I20260620 19:59:21.367153 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.345484 9082 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:21.366560 6376 raft_consensus.cc:2410] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260620 19:59:21.288519 9156 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.368464 6877 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:21.373272 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:21.376592 8730 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.376724 9158 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:21.377254 9158 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:21.378925 9025 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:21.379321 9076 consensus_meta.cc:287] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.010s user 0.008s sys 0.001s
I20260620 19:59:21.379415 9158 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:59:21.379839 9076 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:21.380434 9157 log.cc:927] Time spent T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260620 19:59:21.345937 9074 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.371317 9082 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.380024 9158 consensus_queue.cc:1059] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:21.374480 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:21.389312 6877 leader_election.cc:336] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.902s (SENT)
W20260620 19:59:21.352267 9124 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.016s sys 0.049s
I20260620 19:59:21.383564 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 447) took 1132 ms. Trace:
I20260620 19:59:21.382797 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 443) took 1212 ms. Trace:
W20260620 19:59:21.390977 6877 leader_election.cc:336] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.024s (SENT)
I20260620 19:59:21.395298 6877 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:21.395123 6231 rpcz_store.cc:276] 0620 19:59:20.170088 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.230835 (+ 60747us) service_pool.cc:224] Handling call
0620 19:59:20.231185 (+ 350us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:20.232715 (+ 1530us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:20.232741 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:20.240004 (+ 7263us) log.cc:844] Serialized 52 byte log entry
0620 19:59:21.056502 (+816498us) spinlock_profiling.cc:243] Waited 6.09 ms on lock 0x7b18000e8300. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b8459471 00007fa6b8875def
0620 19:59:21.286242 (+229740us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:21.286303 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.286403 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.382497 (+ 96094us) raft_consensus.cc:1685] finished
0620 19:59:21.382517 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.382738 (+ 221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":493,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":4959,"spinlock_wait_cycles":14047360,"thread_start_us":229494,"threads_started":1,"wal-append.queue_time_us":1137230}
I20260620 19:59:21.368434 8901 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.397396 8920 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.413841 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1659) took 1745 ms. Trace:
I20260620 19:59:21.419589 8920 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.419829 6377 rpcz_store.cc:276] 0620 19:59:19.667912 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.806216 (+138304us) service_pool.cc:224] Handling call
0620 19:59:19.824599 (+ 18383us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:19.933036 (+108437us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:19.933079 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:19.933609 (+ 530us) log.cc:844] Serialized 26 byte log entry
0620 19:59:20.045684 (+112075us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:20.045720 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:20.045785 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.375935 (+1330150us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b180044bf60. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b5796aa6 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b5796aa6 00007fa6b5796a40
0620 19:59:21.376731 (+ 796us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b180044bf60. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b5796aa6 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1
0620 19:59:21.377991 (+ 1260us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b180044bf60. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:21.413589 (+ 35598us) raft_consensus.cc:1685] finished
0620 19:59:21.413617 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.413767 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27452,"raft.queue_time_us":350718,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":3788,"spinlock_wait_cycles":28066816,"thread_start_us":215378,"threads_started":2,"wal-append.queue_time_us":248442}
I20260620 19:59:21.414043 9124 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:21.369774 9005 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.412861 9076 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.395009 6224 rpcz_store.cc:276] 0620 19:59:20.251450 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.117577 (+866127us) service_pool.cc:224] Handling call
0620 19:59:21.117890 (+ 313us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:21.131797 (+ 13907us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:21.131835 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:21.132284 (+ 449us) log.cc:844] Serialized 26 byte log entry
0620 19:59:21.213394 (+ 81110us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:21.213430 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.213493 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.383366 (+169873us) raft_consensus.cc:1685] finished
0620 19:59:21.383402 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.383513 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5563,"raft.queue_time_us":17988,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":12438,"spinlock_wait_cycles":1180416,"thread_start_us":61212,"threads_started":1,"wal-append.queue_time_us":127279}
I20260620 19:59:21.411499 8901 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:21.428710 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1628) took 2600 ms. Trace:
I20260620 19:59:21.429050 6305 rpcz_store.cc:276] 0620 19:59:18.828670 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:18.868935 (+ 40265us) service_pool.cc:224] Handling call
0620 19:59:18.869206 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:19.791007 (+921801us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.428513 (+1637506us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54001a9854. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:21.428659 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2032115584}
W20260620 19:59:21.431178 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.431517 9010 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:21.432039 9010 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.440095 6904 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:21.440718 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 872) took 1002 ms. Trace:
I20260620 19:59:21.441090 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1856) took 1012 ms. Trace:
I20260620 19:59:21.441622 6306 rpcz_store.cc:276] 0620 19:59:20.428205 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.068816 (+640611us) service_pool.cc:224] Handling call
0620 19:59:21.069048 (+ 232us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:21.189943 (+120895us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.440876 (+250933us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54003a6954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:21.441033 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":245105024}
I20260620 19:59:21.248978 6223 rpcz_store.cc:276] 0620 19:59:20.169522 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.172964 (+ 3442us) service_pool.cc:224] Handling call
0620 19:59:20.232458 (+ 59494us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:20.238163 (+ 5705us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:20.240059 (+ 1896us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:20.256543 (+ 16484us) log.cc:844] Serialized 52 byte log entry
0620 19:59:21.146819 (+890276us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:21.159221 (+ 12402us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.165684 (+ 6463us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.248178 (+ 82494us) raft_consensus.cc:1685] finished
0620 19:59:21.248203 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.248403 (+ 200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14123,"raft.queue_time_us":2463,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":9670,"thread_start_us":879953,"threads_started":1,"wal-append.queue_time_us":894402}
I20260620 19:59:21.446550 6227 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:21.448966 6195 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:21.449777 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.450313 6376 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:21.429052 6229 raft_consensus.cc:1275] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260620 19:59:21.451828 9082 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.452615 6229 raft_consensus.cc:1217] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[5.6-5.6] Dedup: 5.6->[]
I20260620 19:59:21.453260 6229 raft_consensus.cc:1240] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260620 19:59:21.454094 9082 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:21.455865 6224 raft_consensus.cc:1240] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:21.440994 6904 rpcz_store.cc:276] 0620 19:59:20.438118 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.483028 (+ 44910us) service_pool.cc:224] Handling call
0620 19:59:21.440641 (+957613us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:21.464241 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 87967us
I20260620 19:59:21.464150 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.466756 6231 raft_consensus.cc:1275] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:21.469918 6271 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
W20260620 19:59:21.474989 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.475670 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.475765 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:21.476783 6376 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:21.446579 9065 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.455191 8954 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:21.499080 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.499145 8954 consensus_queue.cc:1059] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:21.448191 9074 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:21.501367 9158 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:21.502063 9158 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.489213 8985 consensus_queue.cc:1059] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260620 19:59:21.504879 9158 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:21.505631 9158 raft_consensus.cc:3055] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:21.506062 9158 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:21.490435 8999 consensus_queue.cc:1059] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260620 19:59:21.489691 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.466351 9076 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:21.490316 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.492329 8920 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:21.492585 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.514472 6302 raft_consensus.cc:2393] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:21.514808 6304 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:21.497371 9170 raft_consensus.cc:3055] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:21.517848 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.500164 6298 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:21.518574 6377 raft_consensus.cc:3060] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.520722 9010 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:21.521270 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 130290us
W20260620 19:59:21.539206 6877 leader_election.cc:336] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.024s (SENT)
W20260620 19:59:21.526201 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.527141 9158 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.531698 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:21.541853 6304 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 7
W20260620 19:59:21.540130 6877 leader_election.cc:336] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.143s (SENT)
W20260620 19:59:21.544553 6225 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 440) took 1375 ms (client timeout 1558 ms). Trace:
W20260620 19:59:21.544939 6225 rpcz_store.cc:269] 0620 19:59:20.169102 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.169206 (+ 104us) service_pool.cc:224] Handling call
0620 19:59:21.544409 (+1375203us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b5400285d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:21.544501 (+ 92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2061930240}
I20260620 19:59:21.545943 8999 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:21.546237 6225 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260620 19:59:21.532975 6874 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:21.554085 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.489535 8969 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:21.563135 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1657) took 1895 ms. Trace:
I20260620 19:59:21.562470 6381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2004) took 2103 ms. Trace:
W20260620 19:59:21.564683 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 94565.7us
W20260620 19:59:21.567418 6344 leader_election.cc:336] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.116s (SENT)
W20260620 19:59:21.567504 6877 leader_election.cc:336] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.212s (SENT)
I20260620 19:59:21.568195 6877 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:21.569576 6344 leader_election.cc:336] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.527s (SENT)
I20260620 19:59:21.565011 6381 rpcz_store.cc:276] 0620 19:59:19.459204 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.318199 (+1858995us) service_pool.cc:224] Handling call
0620 19:59:21.344690 (+ 26491us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:21.426941 (+ 82251us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:21.426975 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:21.427044 (+ 69us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:21.427080 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.562239 (+135159us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b54000456d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:21.562287 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.562417 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187476864}
I20260620 19:59:21.563591 6379 rpcz_store.cc:276] 0620 19:59:19.667428 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:19.781412 (+113984us) service_pool.cc:224] Handling call
0620 19:59:19.781683 (+ 271us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:19.992139 (+210456us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:19.992170 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:20.043352 (+ 51182us) log.cc:844] Serialized 26 byte log entry
0620 19:59:21.547733 (+1504381us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18004403e0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b846185c 00007fa6b57abc8b 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b8459471
0620 19:59:21.547876 (+ 143us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:21.547911 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.547998 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.548037 (+ 39us) raft_consensus.cc:1685] finished
0620 19:59:21.548048 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.563082 (+ 15034us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":294010,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":47818,"spinlock_wait_cycles":25565952,"thread_start_us":260541,"threads_started":2,"wal-append.queue_time_us":142190}
W20260620 19:59:21.571605 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.572518 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 102650us
I20260620 19:59:21.574618 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.574931 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.576020 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.577322 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:21.546398 8999 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:21.517356 9170 raft_consensus.cc:740] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:59:21.590442 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 57590.4us
I20260620 19:59:21.478973 8960 consensus_queue.cc:1048] T 7a566dbaea0746f4bc76e04b5072e045 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
W20260620 19:59:21.587570 6373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 526) took 2109 ms (client timeout 1693 ms). Trace:
I20260620 19:59:21.478960 8730 consensus_queue.cc:1059] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260620 19:59:21.588966 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.590549 6225 raft_consensus.cc:1240] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:21.550616 9121 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:21.587764 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.594323 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.516705 9005 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:21.593598 6373 rpcz_store.cc:269] 0620 19:59:19.458672 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.567609 (+2108937us) service_pool.cc:211] Skipping call since client already timed out
0620 19:59:21.567715 (+ 106us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260620 19:59:21.599606 6379 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:21.553148 9156 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:21.614712 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.616437 6375 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:21.625372 6379 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
W20260620 19:59:21.629276 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.630532 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.576126 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:21.616835 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.635874 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 68395us
I20260620 19:59:21.591006 8999 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:21.592334 9158 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.593001 9010 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.654464 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.643375 6229 raft_consensus.cc:1275] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260620 19:59:21.643929 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.655520 6229 pending_rounds.cc:85] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 4
I20260620 19:59:21.655787 6229 pending_rounds.cc:107] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260620 19:59:21.651355 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.656018 6229 raft_consensus.cc:2889] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260620 19:59:21.641925 6877 connection.cc:256] RPC call timeout handler was delayed by -1.0150569830000222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
I20260620 19:59:21.636220 6373 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:21.597054 9082 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:21.646716 6377 raft_consensus.cc:2468] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
W20260620 19:59:21.663988 6877 leader_election.cc:336] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.218s (SENT)
W20260620 19:59:21.664999 6877 leader_election.cc:336] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.598s (SENT)
I20260620 19:59:21.622135 6304 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:21.636595 9180 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:21.588194 8969 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.623863 9173 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:21.645047 9156 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:21.679544 9170 consensus_queue.cc:260] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.680176 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.624325 9121 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:21.681509 9165 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.190s user 0.001s sys 0.000s
I20260620 19:59:21.653913 9096 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:21.725426 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.669993 9029 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:21.729249 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.729470 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.737432 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:21.740463 6877 leader_election.cc:336] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 3.866s (SENT)
I20260620 19:59:21.740983 6877 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:21.743841 6271 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:21.640569 9176 consensus_queue.cc:1059] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.106s
I20260620 19:59:21.675783 9180 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:21.750360 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.750919 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.751309 6381 raft_consensus.cc:2393] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:21.751838 6908 raft_consensus.cc:2410] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:21.747411 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.752872 6375 raft_consensus.cc:2368] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8.
I20260620 19:59:21.750895 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.753923 6911 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:21.667595 9010 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:21.762056 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 170785us
I20260620 19:59:21.762751 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.765511 6381 raft_consensus.cc:2393] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:21.660373 9175 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.095s
W20260620 19:59:21.741006 8975 consensus_meta.cc:287] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.522s user 0.010s sys 0.000s
I20260620 19:59:21.770659 9191 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:21.771010 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.771342 9191 consensus_queue.cc:1059] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:21.771692 6902 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:21.772078 9191 consensus_queue.cc:1243] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260620 19:59:21.767817 6877 leader_election.cc:336] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 4.141s (SENT)
I20260620 19:59:21.772671 6877 leader_election.cc:304] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:21.747967 8960 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:21.773563 8960 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.774135 6906 raft_consensus.cc:1275] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:21.775614 8960 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:21.776044 8975 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:21.776137 8960 raft_consensus.cc:3055] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:21.748288 8985 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:21.727629 9156 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.789666 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.790863 6381 raft_consensus.cc:2410] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260620 19:59:21.794620 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 757) took 1658 ms. Trace:
I20260620 19:59:21.750512 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.800580 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.801239 6376 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.754776 9062 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:21.803097 9172 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260620 19:59:21.786547 8960 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:21.784883 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:21.848831 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.816871 6228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1942) took 1245 ms. Trace:
I20260620 19:59:21.795138 6910 rpcz_store.cc:276] 0620 19:59:20.136494 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.136673 (+ 179us) service_pool.cc:224] Handling call
0620 19:59:20.137152 (+ 479us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:20.159612 (+ 22460us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:20.159643 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:20.160228 (+ 585us) log.cc:844] Serialized 52 byte log entry
0620 19:59:21.584248 (+1424020us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b1800418ba0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b8459471 00007fa6b8875def
0620 19:59:21.750636 (+166388us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:21.750714 (+ 78us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.750894 (+ 180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.781377 (+ 30483us) raft_consensus.cc:1685] finished
0620 19:59:21.781396 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.794518 (+ 13122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41428,"raft.queue_time_us":26773,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":18204,"spinlock_wait_cycles":27282688,"thread_start_us":165776,"threads_started":1,"wal-append.queue_time_us":1588230}
W20260620 19:59:21.813380 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 294236us
I20260620 19:59:21.847318 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:21.864178 6903 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:21.870121 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:21.872862 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.873924 6380 raft_consensus.cc:2410] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260620 19:59:21.839053 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:21.765632 9124 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:21.741681 9096 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:21.826941 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.769606 9082 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:21.855986 8960 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:21.856174 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:21.895943 6378 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
W20260620 19:59:21.857609 9182 log.cc:927] Time spent T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260620 19:59:21.858744 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:21.909958 6905 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:21.832963 9165 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260620 19:59:21.861188 9010 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:21.817906 6376 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:21.865118 6228 rpcz_store.cc:276] 0620 19:59:20.571560 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.213293 (+641733us) service_pool.cc:224] Handling call
0620 19:59:21.258881 (+ 45588us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:21.296149 (+ 37268us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:21.298398 (+ 2249us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:21.339244 (+ 40846us) log.cc:844] Serialized 26 byte log entry
0620 19:59:21.456418 (+117174us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:21.456475 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.456588 (+ 113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.816658 (+360070us) raft_consensus.cc:1685] finished
0620 19:59:21.816696 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:21.816819 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57058,"raft.queue_time_us":32625,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":31088,"thread_start_us":83797,"threads_started":1,"wal-append.queue_time_us":189023}
I20260620 19:59:21.892215 6381 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:21.805608 8985 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:21.941867 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.942420 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.942756 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:21.948487 6910 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
W20260620 19:59:21.992386 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.993091 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:21.980417 9114 consensus_meta.cc:287] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.792s user 0.011s sys 0.000s
W20260620 19:59:22.047017 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.050650 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.084219 8969 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.088452 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 879) took 1037 ms. Trace:
W20260620 19:59:22.088986 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 197325us
I20260620 19:59:22.088984 6907 rpcz_store.cc:276] 0620 19:59:21.050954 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.084561 (+ 33607us) service_pool.cc:224] Handling call
0620 19:59:21.084963 (+ 402us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:21.122944 (+ 37981us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:21.122972 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:21.127823 (+ 4851us) log.cc:844] Serialized 26 byte log entry
0620 19:59:21.273557 (+145734us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:21.310527 (+ 36970us) log.cc:844] Serialized 14 byte log entry
0620 19:59:21.322991 (+ 12464us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:21.323061 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:21.493256 (+170195us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b180015bf80. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8d22 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:22.061260 (+568004us) raft_consensus.cc:1685] finished
0620 19:59:22.079024 (+ 17764us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:22.088393 (+ 9369us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19445,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":6867,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":24384,"spinlock_wait_cycles":4068224,"thread_start_us":134501,"threads_started":1,"wal-append.queue_time_us":252700}
I20260620 19:59:22.092737 6196 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:22.095774 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.091801 6875 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.129:36725 timed out after 1.558s (SENT) [suppressed 28 similar messages]
W20260620 19:59:22.120292 6875 leader_election.cc:336] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Timed out: RequestConsensusVote RPC to 127.4.244.129:36725 timed out after 1.780s (SENT)
I20260620 19:59:22.120919 6875 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:22.128759 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:22.131306 6905 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:21.910010 9124 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.138355 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:22.140573 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 355676us
I20260620 19:59:22.143843 6376 raft_consensus.cc:3055] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:22.144062 6380 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:22.144307 6376 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:22.145327 6376 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.146498 6376 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:22.122727 9076 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:22.127729 9010 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:22.134256 8864 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:22.129135 9025 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:22.160859 6301 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1846) took 1793 ms. Trace:
I20260620 19:59:22.161369 6301 rpcz_store.cc:276] 0620 19:59:20.367023 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.367922 (+ 899us) service_pool.cc:224] Handling call
0620 19:59:21.070534 (+702612us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:21.094234 (+ 23700us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:22.160600 (+1066366us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b5400332550. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:22.160784 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1589009792}
I20260620 19:59:22.165652 9156 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:22.167578 9181 log.cc:927] Time spent T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.308s user 0.000s sys 0.001s
W20260620 19:59:22.170024 9196 log.cc:927] Time spent T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.316s user 0.000s sys 0.001s
I20260620 19:59:22.171769 9211 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:22.172269 9211 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.173466 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.174248 9211 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.175892 8985 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:21.894976 9180 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:22.190544 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 324970us
I20260620 19:59:22.190876 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:22.192368 6378 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:22.200696 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.069033 9114 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.215660 6376 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260620 19:59:22.323035 6376 pending_rounds.cc:85] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 5
I20260620 19:59:22.323364 6376 pending_rounds.cc:107] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260620 19:59:22.323652 6376 raft_consensus.cc:2889] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:22.158118 8864 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:22.258424 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.164310 9076 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.315353 9206 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:22.315981 6224 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 7. Current term is 8. Ops: []
W20260620 19:59:22.316311 9193 log.cc:927] Time spent T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.183s user 0.000s sys 0.000s
I20260620 19:59:22.316996 9096 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.323947 6376 pending_rounds.cc:107] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 6.7
W20260620 19:59:22.226879 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.333946 6376 raft_consensus.cc:2889] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:22.328823 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:22.324457 8753 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:22.334859 6378 raft_consensus.cc:2376] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7. Re-sending same reply.
W20260620 19:59:22.338366 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 196593us
I20260620 19:59:22.329530 8864 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:22.339133 6271 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:22.189522 9025 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:22.370720 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.376600 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 252218us
I20260620 19:59:22.373693 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:22.379439 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:22.382210 6196 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:22.384971 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.385545 6378 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:22.410562 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:22.419557 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.386487 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 194331us
W20260620 19:59:22.440011 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.403712 6303 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.001s sys 0.013s
I20260620 19:59:22.448005 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:22.447165 6303 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.002s sys 0.014s
I20260620 19:59:22.375182 8808 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:22.452438 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.455173 6376 raft_consensus.cc:2393] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:59:22.456589 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 117513us
I20260620 19:59:22.456779 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.457731 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:22.460357 8993 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:22.460515 9217 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:22.460917 8993 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.461030 9217 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.340934 8753 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.476603 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:22.480090 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.480482 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.479246 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:22.487856 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.466960 9206 raft_consensus.cc:2804] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:22.488831 6376 raft_consensus.cc:2410] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260620 19:59:22.473482 9197 log.cc:927] Time spent T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.415s user 0.002s sys 0.002s
I20260620 19:59:22.489988 6378 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:22.486850 8993 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.471058 9210 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:22.477588 8985 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:22.481668 9135 raft_consensus.cc:2804] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:22.497531 9216 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:22.498435 9202 log.cc:927] Time spent T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260620 19:59:22.499485 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:22.459910 8808 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.510066 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:22.511070 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.514945 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:22.515568 6907 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:22.517146 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:22.517834 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.517980 6374 raft_consensus.cc:2410] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 7 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:22.517750 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:22.519825 6378 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:22.483152 9209 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260620 19:59:22.521481 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.524401 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.526317 8808 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.525085 6374 raft_consensus.cc:2376] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7. Re-sending same reply.
I20260620 19:59:22.502429 8985 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:22.533650 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.550071 6903 consensus_meta.cc:287] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.685s user 0.010s sys 0.001s
I20260620 19:59:22.550464 6903 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:22.516103 9135 raft_consensus.cc:697] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:22.554015 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:22.531318 9029 consensus_meta.cc:287] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.786s user 0.007s sys 0.004s
I20260620 19:59:22.548349 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:22.560405 6374 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:22.561986 6232 raft_consensus.cc:2410] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260620 19:59:22.501189 9210 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.531802 9217 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.551137 8985 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:22.572465 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:22.577171 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.579440 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.555617 9076 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.513638 9216 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:22.589341 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:22.589658 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 207485us
W20260620 19:59:22.595458 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.567250 9041 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:22.562417 9124 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:22.609426 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 229013us
W20260620 19:59:22.609162 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 145886us
I20260620 19:59:22.646399 6198 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:22.588559 8753 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:22.659613 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.663954 6198 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:22.656342 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 714) took 1181 ms. Trace:
I20260620 19:59:22.668596 9206 raft_consensus.cc:697] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [term 7 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:23.789131 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 1.12503e+06us
I20260620 19:59:22.870831 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:22.907040 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:23.790473 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 919536us
I20260620 19:59:23.322566 9005 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:23.791178 8864 consensus_meta.cc:287] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.412s user 0.010s sys 0.002s
W20260620 19:59:22.668385 9201 log.cc:927] Time spent T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.217s user 0.006s sys 0.002s
W20260620 19:59:22.653532 9197 log.cc:927] Time spent T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260620 19:59:23.791617 8864 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.653848 9041 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:23.214918 6908 raft_consensus.cc:1275] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260620 19:59:22.686007 9203 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:22.840616 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:22.655411 9209 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:23.793758 9210 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:23.794157 8864 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:23.794477 9209 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:23.222579 9220 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:23.796939 9209 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:22.788880 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 762) took 1146 ms. Trace:
W20260620 19:59:23.315227 6227 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1953) took 2234 ms (client timeout 1700 ms). Trace:
I20260620 19:59:22.723488 6303 rpcz_store.cc:276] 0620 19:59:21.474438 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.474563 (+ 125us) service_pool.cc:224] Handling call
0620 19:59:21.474864 (+ 301us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:21.475066 (+ 202us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:21.583275 (+108209us) log.cc:844] Serialized 14 byte log entry
0620 19:59:22.327670 (+744395us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:22.563205 (+235535us) log.cc:844] Serialized 14 byte log entry
0620 19:59:22.656009 (+ 92804us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:22.656071 (+ 62us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:22.656120 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:22.656183 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:22.656288 (+ 105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82522,"prepare.queue_time_us":419097,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":461458,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":6593,"spinlock_wait_cycles":76689664,"thread_start_us":1142895,"threads_started":5,"wal-append.queue_time_us":360225}
W20260620 19:59:23.228929 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:23.293313 9191 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:23.800685 9191 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:23.801420 8864 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:23.802073 8864 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.723956 9175 raft_consensus.cc:3055] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Stepping down as leader of term 7
W20260620 19:59:22.961653 6226 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 439) took 2792 ms (client timeout 1780 ms). Trace:
W20260620 19:59:22.871412 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.912982 9195 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:23.323916 9180 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:22.714500 6378 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
W20260620 19:59:23.803881 6226 rpcz_store.cc:269] 0620 19:59:20.168833 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.168941 (+ 108us) service_pool.cc:224] Handling call
0620 19:59:22.961396 (+2792455us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001414d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:22.961550 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2435641728}
I20260620 19:59:23.796314 9220 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:22.725095 6232 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 1940) took 2258 ms (client timeout 1594 ms). Trace:
W20260620 19:59:23.280613 6877 leader_election.cc:336] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.622s (SENT)
W20260620 19:59:23.809747 6232 rpcz_store.cc:269] 0620 19:59:20.467152 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:20.697573 (+230421us) service_pool.cc:224] Handling call
0620 19:59:22.724769 (+2027196us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54000edb54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:22.724987 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3550679808}
W20260620 19:59:23.118549 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:23.793771 9203 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:22.791035 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:23.792292 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:22.853592 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:23.812076 6910 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:23.813102 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 777) took 1022 ms. Trace:
I20260620 19:59:23.797698 6909 rpcz_store.cc:276] 0620 19:59:21.642985 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.643161 (+ 176us) service_pool.cc:224] Handling call
0620 19:59:21.666704 (+ 23543us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:21.666963 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 7
0620 19:59:22.046190 (+379227us) log.cc:844] Serialized 14 byte log entry
0620 19:59:22.788523 (+742333us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:22.788577 (+ 54us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:22.788622 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:22.788695 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:22.788815 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":353368,"prepare.queue_time_us":72097,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":2835,"raft.queue_time_us":313554,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":23565,"thread_start_us":780763,"threads_started":2,"wal-append.queue_time_us":539340}
W20260620 19:59:23.798524 6227 rpcz_store.cc:269] 0620 19:59:21.081166 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.104302 (+ 23136us) service_pool.cc:224] Handling call
0620 19:59:23.314985 (+2210683us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b540004c9d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:23.315124 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":760760192}
W20260620 19:59:23.791931 9189 log.cc:927] Time spent T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 1.000s user 0.020s sys 0.008s
I20260620 19:59:23.806178 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 566) took 1351 ms. Trace:
I20260620 19:59:22.792906 9222 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:23.794082 6907 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
W20260620 19:59:23.153631 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:23.813910 6910 rpcz_store.cc:276] 0620 19:59:22.790418 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.790742 (+ 324us) service_pool.cc:224] Handling call
0620 19:59:23.812969 (+1022227us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:23.821108 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:23.812755 6905 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
W20260620 19:59:23.898854 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 108622us
W20260620 19:59:23.875138 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:23.411754 6376 raft_consensus.cc:1240] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
W20260620 19:59:23.838506 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 46806.1us
I20260620 19:59:23.975756 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 558) took 1829 ms. Trace:
I20260620 19:59:23.976344 6376 rpcz_store.cc:276] 0620 19:59:22.145961 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:23.410972 (+1265011us) service_pool.cc:224] Handling call
0620 19:59:23.411421 (+ 449us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:23.975493 (+564072us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:23.975701 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:23.976858 6378 rpcz_store.cc:276] 0620 19:59:22.455302 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.557200 (+101898us) service_pool.cc:224] Handling call
0620 19:59:23.806114 (+1248914us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:22.903097 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:23.812196 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 776) took 1022 ms. Trace:
W20260620 19:59:23.129436 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 484138us
W20260620 19:59:23.674767 9158 consensus_meta.cc:287] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 2.016s user 0.014s sys 0.000s
I20260620 19:59:23.319157 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1950) took 2247 ms. Trace:
I20260620 19:59:23.319918 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 722) took 1828 ms. Trace:
I20260620 19:59:23.988863 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 780) took 1140 ms. Trace:
I20260620 19:59:23.989979 6907 rpcz_store.cc:276] 0620 19:59:22.834225 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.834416 (+ 191us) service_pool.cc:224] Handling call
0620 19:59:23.974478 (+1140062us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:22.725240 9213 log.cc:927] Time spent T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
W20260620 19:59:22.971711 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:23.974453 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:23.991444 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:23.992455 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:23.988464 6224 rpcz_store.cc:276] 0620 19:59:21.072080 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.466420 (+394340us) service_pool.cc:224] Handling call
0620 19:59:21.466699 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:22.331283 (+864584us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:23.318870 (+987587us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b5400286754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:23.319085 (+ 215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1952228224}
W20260620 19:59:22.724762 9218 log.cc:927] Time spent T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260620 19:59:23.987473 6273 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:23.990796 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 778) took 1163 ms. Trace:
I20260620 19:59:24.005605 6905 rpcz_store.cc:276] 0620 19:59:22.811118 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.811453 (+ 335us) service_pool.cc:224] Handling call
0620 19:59:23.974361 (+1162908us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:23.986150 6902 rpcz_store.cc:276] 0620 19:59:22.790051 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.790653 (+ 602us) service_pool.cc:224] Handling call
0620 19:59:23.812138 (+1021485us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:23.325137 8920 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:23.497466 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.814299 9227 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:23.978088 6903 raft_consensus.cc:2393] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:22.701637 9212 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.228s user 0.001s sys 0.000s
W20260620 19:59:23.826558 9133 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.001s sys 0.021s
W20260620 19:59:23.876049 9228 log.cc:927] Time spent T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 1.023s user 0.002s sys 0.009s
I20260620 19:59:22.963338 9207 raft_consensus.cc:1064] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661: attempting to promote NON_VOTER 20bd146dc0cb4917a1a5ce05ec609ce1 to VOTER
I20260620 19:59:23.803387 9175 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:59:23.991492 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:22.666764 9029 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:22.926962 8999 consensus_meta.cc:287] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 1.266s user 0.008s sys 0.004s
I20260620 19:59:23.320274 8829 raft_consensus.cc:3055] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:22.724426 9206 consensus_queue.cc:237] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:23.867488 9208 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:24.113523 9208 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:24.113927 9208 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:24.008381 9025 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.008466 9220 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:22.912345 9217 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:22.723774 9115 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:23.819437 9171 raft_consensus.cc:3055] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 19:59:23.989332 6299 rpcz_store.cc:276] 0620 19:59:21.491434 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.516169 (+ 24735us) service_pool.cc:224] Handling call
0620 19:59:21.533064 (+ 16895us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:21.739210 (+206146us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:21.739240 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:21.739289 (+ 49us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:21.739322 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:23.319666 (+1580344us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54001f6154. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:23.319722 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:23.319853 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473571072}
I20260620 19:59:23.221508 9216 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:23.861876 9219 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:23.891667 9099 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:23.159632 9076 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.068264 6379 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:22.844587 9232 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:23.878651 9005 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:24.070747 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 781) took 1216 ms. Trace:
I20260620 19:59:24.134918 6903 rpcz_store.cc:276] 0620 19:59:22.854551 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.854794 (+ 243us) service_pool.cc:224] Handling call
0620 19:59:24.070654 (+1215860us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:24.072337 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 95530.3us
W20260620 19:59:24.074687 6225 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.008s sys 0.020s
I20260620 19:59:23.804608 9195 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.138744 9234 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.139251 9195 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.134718 9076 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.139987 9195 consensus_queue.cc:1059] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:24.141319 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 765) took 2471 ms. Trace:
I20260620 19:59:24.141476 9076 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.142490 9203 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.142715 9025 raft_consensus.cc:3055] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:24.141697 6904 rpcz_store.cc:276] 0620 19:59:21.669696 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.669819 (+ 123us) service_pool.cc:224] Handling call
0620 19:59:21.670127 (+ 308us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:21.837917 (+167790us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:21.837969 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:21.855589 (+ 17620us) log.cc:844] Serialized 52 byte log entry
0620 19:59:22.153552 (+297963us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:22.165238 (+ 11686us) log.cc:844] Serialized 14 byte log entry
0620 19:59:22.165888 (+ 650us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:22.165960 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:24.141099 (+1975139us) raft_consensus.cc:1685] finished
0620 19:59:24.141117 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.141244 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8887,"prepare.queue_time_us":11763,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":54593,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":10969,"thread_start_us":364555,"threads_started":2,"wal-append.queue_time_us":609350}
I20260620 19:59:24.143055 9232 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:24.143828 6303 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260620 19:59:24.140733 9195 consensus_queue.cc:1048] T 64d372664b48432eafee6464408fcef5 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20260620 19:59:24.123836 9237 log.cc:927] Time spent T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20260620 19:59:24.103807 9210 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260620 19:59:23.987190 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 706473us
I20260620 19:59:23.877338 9209 consensus_queue.cc:1048] T 92212144534e4394844003016ba67a47 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.193s
I20260620 19:59:24.107934 9175 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:24.072705 9229 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.004s sys 0.022s
W20260620 19:59:24.073051 9128 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.002s sys 0.025s
I20260620 19:59:24.109054 8999 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.105185 8829 raft_consensus.cc:740] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:24.073933 9133 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s user 0.005s sys 0.025s
I20260620 19:59:22.667174 9135 consensus_queue.cc:237] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:23.042716 9223 consensus_queue.cc:1059] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 1.062s
I20260620 19:59:24.075472 9238 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:24.135829 6225 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.008s sys 0.020s
I20260620 19:59:24.143766 9186 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:24.073205 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 85797.5us
I20260620 19:59:24.153796 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:24.215240 6877 leader_election.cc:336] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 4.928s (SENT)
I20260620 19:59:24.227464 6877 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:24.134537 9099 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:24.156762 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:24.125209 9171 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:24.148380 9005 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.120405 9233 consensus_queue.cc:1048] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.061s
I20260620 19:59:24.101460 9227 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:24.124961 9217 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.157172 9240 consensus_queue.cc:1048] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260620 19:59:24.163064 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:24.156133 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:24.139243 9234 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:23.989337 9222 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:24.180104 6231 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.114s user 0.016s sys 0.048s
W20260620 19:59:24.297740 6231 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.232s user 0.016s sys 0.048s
I20260620 19:59:24.183581 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 556) took 2038 ms. Trace:
I20260620 19:59:24.298583 6377 rpcz_store.cc:276] 0620 19:59:22.145480 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:23.005943 (+860463us) service_pool.cc:224] Handling call
0620 19:59:23.006336 (+ 393us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:23.006577 (+ 241us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:23.273698 (+267121us) log.cc:844] Serialized 14 byte log entry
0620 19:59:24.183099 (+909401us) log.cc:844] Serialized 14 byte log entry
0620 19:59:24.183281 (+ 182us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:24.183323 (+ 42us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:24.183367 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.183425 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.183530 (+ 105us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1140473,"prepare.run_cpu_time_us":867,"prepare.run_wall_time_us":5062,"raft.queue_time_us":165642,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":42376,"spinlock_wait_cycles":33522432,"thread_start_us":978962,"threads_started":3,"wal-append.queue_time_us":643488}
W20260620 19:59:24.201114 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 199491us
W20260620 19:59:24.155040 9046 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.050s sys 0.058s
I20260620 19:59:24.100831 8920 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.225526 6273 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:24.255924 9158 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.155814 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:24.143081 9025 raft_consensus.cc:740] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:24.179797 9128 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.002s sys 0.025s
W20260620 19:59:24.178061 9229 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s user 0.004s sys 0.022s
W20260620 19:59:24.203848 8985 consensus_meta.cc:287] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 1.598s user 0.012s sys 0.000s
I20260620 19:59:24.147677 9219 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.181265 9175 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:24.272290 9235 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:24.273092 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.288309 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:24.251210 9076 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.250417 9082 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:24.252288 9238 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.448817 6903 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260620 19:59:24.461551 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.369695 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.314502 6374 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.007s sys 0.015s
I20260620 19:59:24.256135 9186 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.296603 9223 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:24.450100 8920 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:24.306936 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 81487.4us
W20260620 19:59:24.378662 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 200954us
I20260620 19:59:24.447657 9232 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:24.298003 9240 consensus_queue.cc:1243] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260620 19:59:24.522670 6374 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.051s user 0.007s sys 0.023s
W20260620 19:59:24.522498 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.531522 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.532827 6273 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:24.465792 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:24.535025 6905 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 7
I20260620 19:59:24.535983 6231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1951) took 3462 ms. Trace:
W20260620 19:59:24.536584 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.496166 8864 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.504200 9005 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:24.307441 6907 raft_consensus.cc:2410] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260620 19:59:24.483560 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 182798us
W20260620 19:59:24.465229 6875 leader_election.cc:336] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725): Timed out: RequestConsensusVote RPC to 127.4.244.129:36725 timed out after 1.558s (SENT)
I20260620 19:59:24.524854 9220 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:24.334184 9239 log.cc:927] Time spent T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260620 19:59:24.532181 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 554) took 2387 ms. Trace:
W20260620 19:59:24.415663 9243 log.cc:927] Time spent T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260620 19:59:24.562228 6374 rpcz_store.cc:276] 0620 19:59:22.144909 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.585307 (+440398us) service_pool.cc:224] Handling call
0620 19:59:22.591743 (+ 6436us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:22.647249 (+ 55506us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:22.647280 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:22.653150 (+ 5870us) log.cc:844] Serialized 26 byte log entry
0620 19:59:23.121909 (+468759us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:24.530976 (+1409067us) log.cc:844] Serialized 14 byte log entry
0620 19:59:24.531903 (+ 927us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.531970 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:24.532006 (+ 36us) raft_consensus.cc:1685] finished
0620 19:59:24.532018 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.532131 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1027125,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":14015,"raft.queue_time_us":39030,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":38532,"thread_start_us":1486134,"threads_started":2,"wal-append.queue_time_us":1340038}
W20260620 19:59:24.451354 9246 log.cc:927] Time spent T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260620 19:59:24.334928 9236 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:24.552014 9203 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:24.525846 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 451) took 3256 ms. Trace:
I20260620 19:59:24.548506 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:24.551306 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.551180 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:24.552829 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 411534us
I20260620 19:59:24.508626 9227 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:24.296675 8829 consensus_queue.cc:260] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:24.526300 9158 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.506919 9099 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:24.553215 6273 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:24.536445 6231 rpcz_store.cc:276] 0620 19:59:21.074180 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.469149 (+394969us) service_pool.cc:224] Handling call
0620 19:59:21.481038 (+ 11889us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:21.512307 (+ 31269us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:21.512349 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:21.512872 (+ 523us) log.cc:844] Serialized 26 byte log entry
0620 19:59:21.732362 (+219490us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:23.065725 (+1333363us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b18003488e0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b574340a 00007fa6bee885d9 00007fa6b57cb240 00007fa6b57c5ce4 00007fa6b57c5c55 00007fa6bc2b4247 00007fa6b88c4d87 00007fa6b88d449f 00007fa6b88d4418
0620 19:59:24.298204 (+1232479us) log.cc:844] Serialized 14 byte log entry
0620 19:59:24.523757 (+225553us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.523826 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:24.523870 (+ 44us) raft_consensus.cc:1685] finished
0620 19:59:24.535729 (+ 11859us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.535915 (+ 186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23084,"prepare.queue_time_us":1436830,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":33182,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":16433,"spinlock_wait_cycles":19934080,"thread_start_us":1621214,"threads_started":3,"wal-append.queue_time_us":214325,"wal-append.run_cpu_time_us":2557,"wal-append.run_wall_time_us":1406819}
W20260620 19:59:24.561790 6877 leader_election.cc:336] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.558s (SENT)
I20260620 19:59:24.579613 6905 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:24.580242 6875 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:24.562645 6198 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:24.524183 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.302403 9046 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.013s user 0.050s sys 0.058s
I20260620 19:59:24.502367 9223 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.483031 8985 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.548306 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.552642 9232 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.454200 9171 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.577634 6376 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:24.592877 9227 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:24.624676 9222 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.600615 8829 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:24.625734 9217 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.482903 9076 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.501242 9235 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.552109 9226 consensus_queue.cc:1048] T ee78433ca73b4ea8b2b24535036f6369 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260620 19:59:24.584100 6198 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:24.634107 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 50150.4us
I20260620 19:59:24.577982 6225 rpcz_store.cc:276] 0620 19:59:21.265447 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.658977 (+393530us) service_pool.cc:224] Handling call
0620 19:59:21.674890 (+ 15913us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:21.675222 (+ 332us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:21.679577 (+ 4355us) log.cc:844] Serialized 14 byte log entry
0620 19:59:22.723461 (+1043884us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b18003affc0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:23.493729 (+770268us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b18003affc0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b84595da 00007fa6b88c4e0a
0620 19:59:24.256005 (+762276us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:24.500643 (+244638us) log.cc:844] Serialized 26 byte log entry
0620 19:59:24.500822 (+ 179us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:24.500868 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.501112 (+ 244us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:24.522056 (+ 20944us) raft_consensus.cc:1685] finished
0620 19:59:24.522073 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.522187 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":446311,"prepare.queue_time_us":6078,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":11427,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":65500,"spinlock_wait_cycles":15756544,"thread_start_us":744648,"threads_started":1,"wal-append.queue_time_us":2570394}
W20260620 19:59:24.591012 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.594635 9186 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.601318 9229 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.641212 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.641820 6374 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:24.643841 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.644524 6374 raft_consensus.cc:2393] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 3: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:24.646013 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.598137 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 785) took 1539 ms. Trace:
I20260620 19:59:24.569319 9220 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:24.580355 9238 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:24.604916 6271 connection.cc:256] RPC call timeout handler was delayed by -1.2003183489999856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
I20260620 19:59:24.653007 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.653208 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:24.653606 6374 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:24.654388 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.654810 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:24.587503 9099 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:24.606801 6377 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:24.663501 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.664116 6377 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:24.588326 9206 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:24.557771 9210 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:24.615429 8985 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.615890 8960 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:24.675580 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.675905 9215 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:24.676369 9215 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:24.676615 6298 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1863) took 2892 ms. Trace:
I20260620 19:59:24.676741 9215 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:24.677016 6298 rpcz_store.cc:276] 0620 19:59:21.784009 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.786329 (+ 2320us) service_pool.cc:224] Handling call
0620 19:59:21.803579 (+ 17250us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:23.888491 (+2084912us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b18004088e0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b846185c 00007fa6b57abc8b 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c5c55 00007fa6b88b2440 00007fa6b88b2fdb 00007fa6b88bda2b
0620 19:59:23.888664 (+ 173us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:24.299085 (+410421us) log.cc:844] Serialized 14 byte log entry
0620 19:59:24.573423 (+274338us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:24.609395 (+ 35972us) log.cc:844] Serialized 26 byte log entry
0620 19:59:24.633336 (+ 23941us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:24.639723 (+ 6387us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.639797 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:24.676288 (+ 36491us) raft_consensus.cc:1685] finished
0620 19:59:24.676393 (+ 105us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.676555 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7755,"prepare.queue_time_us":646045,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":4797,"raft.queue_time_us":267622,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":37124,"spinlock_wait_cycles":27785344,"thread_start_us":700102,"threads_started":3,"wal-append.queue_time_us":160335}
I20260620 19:59:24.615790 9232 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:24.679522 9265 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260620 19:59:24.681128 9232 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.618085 9121 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:24.620440 9171 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
W20260620 19:59:24.622566 9251 log.cc:927] Time spent T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20260620 19:59:24.508985 9207 consensus_queue.cc:237] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.672848 9210 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:24.684032 9210 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:24.623631 9262 consensus_queue.cc:1048] T 779819d94546427fa367040185e417ab P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.033s
I20260620 19:59:24.624378 9227 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.557368 9174 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:24.627624 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.699198 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.581084 9203 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.639070 6376 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260620 19:59:24.701454 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.702131 6376 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:24.646477 9076 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.642094 9236 raft_consensus.cc:697] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:24.718668 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.719718 6378 raft_consensus.cc:2410] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260620 19:59:24.596591 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.648655 6906 rpcz_store.cc:276] 0620 19:59:23.059258 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.597126 (+1537868us) service_pool.cc:224] Handling call
0620 19:59:24.597482 (+ 356us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:24.597718 (+ 236us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:24.597759 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:24.597817 (+ 58us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:24.597864 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.597937 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.598068 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:24.639508 9208 consensus_meta.cc:287] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.519s user 0.006s sys 0.004s
I20260620 19:59:24.655347 6227 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:24.656912 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.660109 9186 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:24.660682 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.661253 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:24.661110 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:24.661895 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.655004 6374 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:24.639920 9180 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:24.672254 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.681653 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.725865 6224 raft_consensus.cc:2393] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:24.717301 9211 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.223s user 0.014s sys 0.039s
I20260620 19:59:24.686863 9158 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.725483 6230 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
W20260620 19:59:24.695557 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.734548 9223 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.735816 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.761353 6225 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:24.650108 9229 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.737514 8969 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.738348 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.739770 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:24.741024 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.763070 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.777581 6374 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:24.723508 8864 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.725220 9186 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:24.769348 9215 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.764714 9203 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.778046 6229 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:24.806419 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:24.765877 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 391647us
I20260620 19:59:24.726363 9135 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.785310 6376 raft_consensus.cc:3055] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:24.808616 6376 raft_consensus.cc:740] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:24.809350 6376 consensus_queue.cc:260] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.810218 6376 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:24.785965 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:24.786350 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.787045 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:24.687772 9206 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.814014 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:24.787961 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.788681 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:24.728392 9219 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:24.789639 8985 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:24.790263 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:24.784022 8969 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:24.727519 9208 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.791713 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.822575 6378 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:24.792106 8864 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:24.783877 8960 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.795436 9186 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:24.783061 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.802963 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:24.757644 9211 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.266s user 0.015s sys 0.039s
I20260620 19:59:24.803929 6227 raft_consensus.cc:2468] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:24.803015 6374 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260620 19:59:24.828497 6374 pending_rounds.cc:85] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 3
I20260620 19:59:24.828740 6374 pending_rounds.cc:107] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260620 19:59:24.829041 6374 raft_consensus.cc:2889] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:24.820976 6305 raft_consensus.cc:1240] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:24.776403 9099 raft_consensus.cc:697] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:24.797272 9275 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:24.834508 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:24.832051 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.849483 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.849715 6373 raft_consensus.cc:3060] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:24.849555 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.835501 6298 raft_consensus.cc:1240] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:24.835904 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.852375 6230 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:24.830885 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:24.833197 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:24.833129 6231 raft_consensus.cc:1240] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:24.887933 6300 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:24.893149 6376 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260620 19:59:24.893718 6376 pending_rounds.cc:85] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 5
I20260620 19:59:24.894057 6376 pending_rounds.cc:107] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260620 19:59:24.897958 6376 raft_consensus.cc:2889] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:24.899513 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.900760 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:24.902304 6376 raft_consensus.cc:1240] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:24.856880 6305 raft_consensus.cc:2393] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:24.907249 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:24.903800 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 377626us
W20260620 19:59:24.909252 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.912809 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:24.913606 6303 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
W20260620 19:59:24.891605 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 93928.8us
I20260620 19:59:24.916185 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 784) took 1877 ms. Trace:
I20260620 19:59:24.916563 6909 rpcz_store.cc:276] 0620 19:59:23.038864 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.915434 (+1876570us) service_pool.cc:224] Handling call
0620 19:59:24.915721 (+ 287us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:24.915893 (+ 172us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:24.915918 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:24.915957 (+ 39us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:24.915988 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.916040 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.916134 (+ 94us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:24.918987 9191 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:24.922415 9114 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:24.927328 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.927811 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.935931 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:24.944676 6374 pending_rounds.cc:85] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 5
I20260620 19:59:24.945003 6374 pending_rounds.cc:107] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260620 19:59:24.945231 6374 raft_consensus.cc:2889] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:24.944272 6130 catalog_manager.cc:5696] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 2 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 5 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:24.952080 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:24.952704 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:24.954896 6303 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260620 19:59:24.955495 6303 pending_rounds.cc:85] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 7
I20260620 19:59:24.955848 6303 pending_rounds.cc:107] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260620 19:59:24.956238 6303 raft_consensus.cc:2889] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:24.958921 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:24.959583 6223 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1946) took 3939 ms. Trace:
I20260620 19:59:24.960137 6223 rpcz_store.cc:276] 0620 19:59:21.020263 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.442995 (+422732us) service_pool.cc:224] Handling call
0620 19:59:22.726739 (+1283744us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:24.177548 (+1450809us) raft_consensus.cc:1534] Early marking committed up to index 3
0620 19:59:24.221738 (+ 44190us) log.cc:844] Serialized 14 byte log entry
0620 19:59:24.475498 (+253760us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:24.501189 (+ 25691us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:24.516220 (+ 15031us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.958321 (+442101us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54000edb54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:24.958397 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:24.959374 (+ 977us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54000edb50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:24.959511 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":787,"prepare.queue_time_us":180,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":283672,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":37700,"spinlock_wait_cycles":5462552192,"thread_start_us":429429,"threads_started":2,"wal-append.queue_time_us":355521}
I20260620 19:59:24.971589 9234 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:24.976356 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.977149 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.977517 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:24.980686 6406 heartbeater.cc:646] Failed to heartbeat to 127.4.244.190:33963 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.4.244.190:33963 timed out after 15.000s (SENT)
I20260620 19:59:24.984925 8999 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:24.989845 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.991286 9170 raft_consensus.cc:3060] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:24.993585 9235 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.000219 9041 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:25.003067 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.004621 8969 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:25.006664 6379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 529) took 3366 ms (client timeout 2218 ms). Trace:
W20260620 19:59:25.007230 6379 rpcz_store.cc:269] 0620 19:59:21.640484 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.654250 (+ 13766us) service_pool.cc:224] Handling call
0620 19:59:25.006371 (+3352121us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b54003c2b54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:25.006524 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5401291008}
I20260620 19:59:25.010824 6381 raft_consensus.cc:1275] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:25.012869 6381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 546) took 2870 ms. Trace:
I20260620 19:59:25.013329 6381 rpcz_store.cc:276] 0620 19:59:22.142802 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:22.142920 (+ 118us) service_pool.cc:224] Handling call
0620 19:59:25.010361 (+2867441us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:25.011337 (+ 976us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:25.012592 (+ 1255us) spinlock_profiling.cc:243] Waited 2.87 s on lock 0x7b54003c2b50. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:25.012767 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6592801664}
W20260620 19:59:25.023586 6344 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.244.190:33963, user_credentials={real_user=slave}} blocked reactor thread for 45442.5us
I20260620 19:59:24.906179 9220 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:25.050688 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 783) took 2077 ms. Trace:
I20260620 19:59:25.051349 6908 rpcz_store.cc:276] 0620 19:59:22.973307 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:25.049598 (+2076291us) service_pool.cc:224] Handling call
0620 19:59:25.050025 (+ 427us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:25.050257 (+ 232us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:25.050293 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:25.050349 (+ 56us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:25.050398 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:25.050465 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:25.050615 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:25.060585 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:25.061672 6223 raft_consensus.cc:2393] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 8: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:25.063444 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:25.069059 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:25.069633 6903 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:25.071069 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:25.073640 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.887221 6226 raft_consensus.cc:2376] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7. Re-sending same reply.
W20260620 19:59:25.077461 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:24.819101 9194 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:25.078644 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 186341us
W20260620 19:59:25.079461 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 614144us
W20260620 19:59:25.097012 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.091722 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:25.085577 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:25.086150 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.105183 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:25.110857 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.115252 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:25.117241 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:24.723827 9174 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:24.856169 9219 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:24.827596 8985 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.127280 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.129768 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.131327 6300 raft_consensus.cc:2410] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260620 19:59:24.825646 9135 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:25.136514 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:25.132509 6198 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:25.134440 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 773) took 3193 ms. Trace:
I20260620 19:59:24.835119 8960 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:25.143348 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:25.151120 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 69623.1us
I20260620 19:59:25.152240 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:24.739583 9236 consensus_queue.cc:237] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:25.158228 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.159996 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.208432 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 100992us
W20260620 19:59:25.208763 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.184226 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.206415 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.209861 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.207024 6378 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
W20260620 19:59:25.219766 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.223766 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:25.118049 9005 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260620 19:59:25.140319 6911 rpcz_store.cc:276] 0620 19:59:21.940936 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.942642 (+ 1706us) service_pool.cc:224] Handling call
0620 19:59:22.089182 (+146540us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:22.164290 (+ 75108us) raft_consensus.cc:1534] Early marking committed up to index 7
0620 19:59:22.324436 (+160146us) log.cc:844] Serialized 14 byte log entry
0620 19:59:23.274748 (+950312us) log.cc:844] Serialized 14 byte log entry
0620 19:59:23.275441 (+ 693us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:23.275496 (+ 55us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:23.275561 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:25.133720 (+1858159us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b5400631ad4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:25.133780 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:25.133944 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":179253,"prepare.queue_time_us":265710,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":4869,"raft.queue_time_us":167513,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":18474,"spinlock_wait_cycles":158925568,"thread_start_us":782760,"threads_started":2,"wal-append.queue_time_us":467068,"wal-append.run_cpu_time_us":14151,"wal-append.run_wall_time_us":2301212}
I20260620 19:59:25.250502 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:25.250718 6345 connection.cc:256] RPC call timeout handler was delayed by -2.6736839480000754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
W20260620 19:59:25.251315 6345 connection.cc:256] RPC call timeout handler was delayed by -2.6736839480000754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
W20260620 19:59:25.251720 6345 connection.cc:256] RPC call timeout handler was delayed by -2.6691839480000681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260620 19:59:25.252089 6345 connection.cc:256] RPC call timeout handler was delayed by -2.6691839480000681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260620 19:59:25.252517 6345 connection.cc:256] RPC call timeout handler was delayed by -2.5575839480000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1675s for a response
W20260620 19:59:25.252934 6345 connection.cc:256] RPC call timeout handler was delayed by -2.5575839480000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1675s for a response
I20260620 19:59:25.253841 6307 raft_consensus.cc:3060] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 5
W20260620 19:59:25.253301 6345 connection.cc:256] RPC call timeout handler was delayed by -2.53598394800008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
W20260620 19:59:25.254756 6345 connection.cc:256] RPC call timeout handler was delayed by -2.53598394800008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
W20260620 19:59:25.255136 6345 connection.cc:256] RPC call timeout handler was delayed by -2.5323839480000743s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
I20260620 19:59:25.211285 6196 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:25.255623 6345 connection.cc:256] RPC call timeout handler was delayed by -2.4963839480000161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1743s for a response
W20260620 19:59:25.260951 6345 connection.cc:256] RPC call timeout handler was delayed by -2.2686839480001026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
W20260620 19:59:25.261411 6345 connection.cc:256] RPC call timeout handler was delayed by -2.2092839480000066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
W20260620 19:59:25.261818 6345 connection.cc:256] RPC call timeout handler was delayed by -2.2092839480000066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
W20260620 19:59:25.262202 6345 connection.cc:256] RPC call timeout handler was delayed by -2.1606839480000417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20260620 19:59:25.262593 6345 connection.cc:256] RPC call timeout handler was delayed by -1.7907839480000121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25270000000000004s for a response
W20260620 19:59:25.263000 6345 connection.cc:256] RPC call timeout handler was delayed by -1.6143839480000679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2723s for a response
I20260620 19:59:25.264015 6345 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:25.130384 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:25.271690 6298 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260620 19:59:25.276386 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.277202 6304 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:25.078425 9275 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.278264 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.268266 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 116087us
I20260620 19:59:25.289932 6304 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:25.153674 9112 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:25.127864 9210 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.257033 9176 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:25.105720 9194 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.126578 9128 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.141s
I20260620 19:59:25.100768 9099 consensus_queue.cc:237] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:25.306373 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.311391 6304 raft_consensus.cc:2410] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260620 19:59:25.145325 9219 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.265870 8954 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:25.266062 9234 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:25.265687 9156 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:25.307924 9174 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:25.312265 6373 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:25.293494 9223 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260620 19:59:25.299320 6377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.084s sys 0.226s
I20260620 19:59:25.301241 9203 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:25.265271 9262 consensus_queue.cc:1048] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260620 19:59:25.298794 9005 consensus_queue.cc:1059] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
W20260620 19:59:25.347702 6377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.084s sys 0.226s
I20260620 19:59:25.325519 6345 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:25.312418 9227 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.363806 9203 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 19:59:25.365603 9005 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:25.367908 9275 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:25.346791 9264 log.cc:927] Time spent T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.619s user 0.000s sys 0.001s
W20260620 19:59:25.326496 9276 log.cc:927] Time spent T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20260620 19:59:25.311203 9112 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.313139 9176 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:25.345407 9229 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.346294 9135 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:25.336066 9215 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.336552 9208 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.347633 9223 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:25.387732 9223 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:25.387810 9005 raft_consensus.cc:3055] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Stepping down as leader of term 2
I20260620 19:59:25.388252 9005 raft_consensus.cc:740] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:25.389103 9005 consensus_queue.cc:260] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.390244 9005 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:25.346709 9206 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.343477 8985 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.345489 9082 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260620 19:59:25.321979 8954 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.393147 8954 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.394968 8960 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:25.396134 9210 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.396605 8864 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.348987 9156 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:25.397373 8829 consensus_meta.cc:287] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.772s user 0.005s sys 0.005s
I20260620 19:59:25.345009 9234 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.398913 9156 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.399310 9234 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.400475 9219 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:25.400995 9112 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:25.396134 9194 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:25.473359 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 261916us
I20260620 19:59:25.475310 6196 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:25.476928 6196 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:25.482869 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 350205us
W20260620 19:59:25.492940 6229 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.003s sys 0.011s
W20260620 19:59:25.493539 6229 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s user 0.003s sys 0.011s
I20260620 19:59:25.507247 6130 catalog_manager.cc:5696] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:59:25.517795 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.518810 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.519456 6271 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:25.521545 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.318735 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.523703 6304 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:25.525220 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.527227 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.527557 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.528000 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.534935 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.536578 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.538524 6374 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.006s sys 0.017s
W20260620 19:59:25.549820 6374 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s user 0.006s sys 0.017s
I20260620 19:59:25.551851 8829 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:25.552654 8829 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:25.553120 8829 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:25.543349 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.546617 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 457009us
I20260620 19:59:25.555255 6300 raft_consensus.cc:2393] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:59:25.539230 6379 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.002s sys 0.023s
W20260620 19:59:25.557729 6379 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s user 0.003s sys 0.023s
W20260620 19:59:25.547133 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.548910 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.551792 8808 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260620 19:59:25.559054 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.559048 8808 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:25.559862 8808 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:25.560599 6298 raft_consensus.cc:2410] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:25.561483 8864 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:25.561996 8864 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:25.548645 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 891) took 1403 ms. Trace:
I20260620 19:59:25.562352 8864 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:25.562569 6904 rpcz_store.cc:276] 0620 19:59:24.145140 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.145473 (+ 333us) service_pool.cc:224] Handling call
0620 19:59:24.145750 (+ 277us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:24.146080 (+ 330us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:24.146121 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:24.146526 (+ 405us) log.cc:844] Serialized 26 byte log entry
0620 19:59:24.386003 (+239477us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:24.386043 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:24.386108 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:25.548024 (+1161916us) raft_consensus.cc:1685] finished
0620 19:59:25.548071 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:25.548362 (+ 291us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2218,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":4986,"thread_start_us":205505,"threads_started":1,"wal-append.queue_time_us":488951}
I20260620 19:59:25.564612 9025 consensus_queue.cc:260] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:25.567484 9025 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:25.572471 8999 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:25.574643 8999 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:25.575327 8999 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:25.596338 9041 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:25.596997 9041 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:25.597461 9041 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:25.631778 9186 consensus_meta.cc:287] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.783s user 0.006s sys 0.003s
I20260620 19:59:25.632330 9186 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.635289 6305 raft_consensus.cc:2393] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 9: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:25.638567 9186 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:25.641649 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.642196 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.642627 9223 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:25.645953 9223 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:25.647665 9222 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:25.647156 6130 catalog_manager.cc:5696] T 93c3314daf2a488caf8f0d3a805f44cd P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 3 to 4, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 4 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
W20260620 19:59:25.649369 6129 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47092 (request call id 378) took 16151 ms (client timeout 15000 ms). Trace:
I20260620 19:59:25.529119 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.650378 6196 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:25.642656 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.806963 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.807587 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.651386 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.829809 6130 catalog_manager.cc:5696] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:59:25.856105 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 205848us
I20260620 19:59:25.651134 9176 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:25.864987 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:25.865999 6903 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:25.868331 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:25.869194 6908 raft_consensus.cc:2368] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5.
I20260620 19:59:25.869977 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:25.871076 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:25.875090 9262 raft_consensus.cc:2804] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:25.875835 9262 raft_consensus.cc:697] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:25.877434 9262 consensus_queue.cc:237] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:25.886108 9170 consensus_meta.cc:287] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.894s user 0.014s sys 0.008s
W20260620 19:59:25.890441 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 406550us
I20260620 19:59:25.911626 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:25.910944 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:25.914127 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:25.914521 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:25.939841 6911 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:25.957816 6305 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:25.960561 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:25.967592 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.974277 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:25.972958 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.975221 9236 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.002s sys 0.009s
W20260620 19:59:25.984768 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:25.980602 9236 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s user 0.002s sys 0.010s
I20260620 19:59:25.991248 6372 raft_consensus.cc:2410] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:26.001148 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:26.001588 6875 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.002s sys 0.011s
I20260620 19:59:26.002156 6305 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 8: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:26.002101 6303 raft_consensus.cc:1240] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:26.003374 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.004784 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.005427 6907 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.006807 9158 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.001344 6903 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEARNER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260620 19:59:26.002110 6875 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s user 0.003s sys 0.011s
I20260620 19:59:25.996995 6909 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.009523 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:25.995780 6302 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.002s sys 0.009s
W20260620 19:59:26.013782 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:26.007341 6306 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.185s user 0.007s sys 0.010s
I20260620 19:59:26.019340 6301 raft_consensus.cc:2410] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260620 19:59:26.024521 6302 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s user 0.003s sys 0.010s
W20260620 19:59:26.027041 6306 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.230s user 0.008s sys 0.010s
I20260620 19:59:26.029388 6306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1652) took 1234 ms. Trace:
W20260620 19:59:26.023900 6877 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.036s sys 0.050s
W20260620 19:59:26.030941 6877 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.036s sys 0.051s
W20260620 19:59:26.031261 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 722482us
W20260620 19:59:26.041471 6224 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.006s sys 0.007s
W20260620 19:59:26.041375 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 60316.7us
W20260620 19:59:26.042071 6224 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s user 0.006s sys 0.007s
W20260620 19:59:26.043109 6377 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.000s sys 0.012s
W20260620 19:59:26.043524 6377 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.689s user 0.000s sys 0.012s
W20260620 19:59:26.041441 6381 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.004s sys 0.008s
I20260620 19:59:26.045096 6377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2022) took 1407 ms. Trace:
W20260620 19:59:26.045121 6381 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.004s sys 0.008s
W20260620 19:59:26.047919 6345 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.003s sys 0.010s
W20260620 19:59:26.048272 6345 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s user 0.003s sys 0.010s
I20260620 19:59:26.048444 6381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1693) took 1229 ms. Trace:
W20260620 19:59:26.049307 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 723721us
W20260620 19:59:26.024027 9099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.054s sys 0.123s
W20260620 19:59:26.051110 9099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.054s sys 0.124s
W20260620 19:59:26.056180 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:26.050643 6373 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.007s sys 0.005s
W20260620 19:59:26.058545 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.050463 6345 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:26.063541 6373 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s user 0.008s sys 0.006s
I20260620 19:59:26.064790 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.045559 6377 rpcz_store.cc:276] 0620 19:59:24.638116 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.665664 (+ 27548us) service_pool.cc:224] Handling call
0620 19:59:24.665924 (+ 260us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:24.666121 (+ 197us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:24.720328 (+ 54207us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.044273 (+1323945us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.044600 (+ 327us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:26.044675 (+ 75us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:26.044747 (+ 72us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.044852 (+ 105us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.045018 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":51951,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":8555,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"thread_start_us":1323108,"threads_started":2,"wal-append.queue_time_us":662744}
I20260620 19:59:26.048870 6381 rpcz_store.cc:276] 0620 19:59:24.819466 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:25.014518 (+195052us) service_pool.cc:224] Handling call
0620 19:59:25.033805 (+ 19287us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:25.034063 (+ 258us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:25.034709 (+ 646us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.048008 (+1013299us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:26.048090 (+ 82us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:26.048168 (+ 78us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.048253 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.048396 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":144,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":29320,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":2407,"thread_start_us":1012485,"threads_started":2,"wal-append.queue_time_us":377563}
I20260620 19:59:26.087069 9005 consensus_queue.cc:1048] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:26.089270 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.090315 6903 raft_consensus.cc:2410] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260620 19:59:26.090710 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.090804 6227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 467) took 1382 ms. Trace:
I20260620 19:59:26.091574 6305 raft_consensus.cc:2393] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 8: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:26.092519 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.091426 6227 rpcz_store.cc:276] 0620 19:59:24.658182 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.830534 (+172352us) service_pool.cc:224] Handling call
0620 19:59:24.830844 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:24.831041 (+ 197us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:25.491269 (+660228us) log.cc:844] Serialized 14 byte log entry
0620 19:59:25.975629 (+484360us) log.cc:844] Serialized 14 byte log entry
0620 19:59:25.983721 (+ 8092us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:25.989664 (+ 5943us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:25.998421 (+ 8757us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.015477 (+ 17056us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.040621 (+ 25144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57176,"prepare.queue_time_us":356183,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":118,"raft.queue_time_us":14564,"raft.run_cpu_time_us":488,"raft.run_wall_time_us":44634,"thread_start_us":831027,"threads_started":2,"wal-append.queue_time_us":571514}
I20260620 19:59:26.095376 9309 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:26.096117 9133 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:59:26.101651 6198 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:26.029932 6306 rpcz_store.cc:276] 0620 19:59:24.795118 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.796013 (+ 895us) service_pool.cc:224] Handling call
0620 19:59:24.796324 (+ 311us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:24.796510 (+ 186us) raft_consensus.cc:1534] Early marking committed up to index 8
0620 19:59:24.796771 (+ 261us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.028403 (+1231632us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.028744 (+ 341us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:26.028829 (+ 85us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:26.028929 (+ 100us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.029066 (+ 137us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.029287 (+ 221us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3294,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":84,"raft.queue_time_us":5311,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":186,"spinlock_wait_cycles":222208,"thread_start_us":1230554,"threads_started":1}
I20260620 19:59:26.104275 6306 raft_consensus.cc:1240] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:26.007347 9158 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.104841 9215 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 19:59:26.064190 8829 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.109500 8829 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.110265 6907 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:26.093532 6305 raft_consensus.cc:2410] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260620 19:59:25.649878 6129 rpcz_store.cc:269] 0620 19:59:09.498287 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:09.559386 (+ 61099us) service_pool.cc:224] Handling call
0620 19:59:25.649072 (+16089686us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:59:17.263357 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:59:17.264341 (+ 984us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:59:17.264416 (+ 75us) write_op.cc:435] Acquired schema lock
0620 19:59:17.264446 (+ 30us) tablet.cc:662] Decoding operations
0620 19:59:17.266523 (+ 2077us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:59:17.314473 (+ 47950us) write_op.cc:641] Partition lock acquired for write op
0620 19:59:17.365971 (+ 51498us) tablet.cc:685] Acquiring locks for 18 operations
0620 19:59:17.476203 (+110232us) tablet.cc:701] Row locks acquired
0620 19:59:17.476229 (+ 26us) write_op.cc:260] PREPARE: finished
0620 19:59:17.476412 (+ 183us) write_op.cc:270] Start()
0620 19:59:17.476566 (+ 154us) write_op.cc:276] Timestamp: P: 1781985557476345 usec, L: 0
0620 19:59:17.476586 (+ 20us) op_driver.cc:348] REPLICATION: starting
0620 19:59:17.477314 (+ 728us) log.cc:844] Serialized 7078 byte log entry
0620 19:59:18.693376 (+1216062us) op_driver.cc:464] REPLICATION: finished
0620 19:59:19.040878 (+347502us) write_op.cc:301] APPLY: starting
0620 19:59:19.041001 (+ 123us) tablet.cc:1367] starting BulkCheckPresence
0620 19:59:19.041363 (+ 362us) tablet.cc:1370] finished BulkCheckPresence
0620 19:59:19.041490 (+ 127us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:59:19.043543 (+ 2053us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:59:19.043902 (+ 359us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0620 19:59:19.043947 (+ 45us) write_op.cc:312] APPLY: finished
0620 19:59:19.044375 (+ 428us) log.cc:844] Serialized 125 byte log entry
0620 19:59:19.119089 (+ 74714us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:19.128281 (+ 9192us) write_op.cc:454] Released schema lock
0620 19:59:19.240426 (+112145us) write_op.cc:341] FINISH: Updating metrics
0620 19:59:19.361696 (+121270us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:19.368733 (+ 7037us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":45526,"reported_tablets":37,"spinlock_wait_cycles":15744,"child_traces":[["op",{"apply.queue_time_us":288144,"apply.run_cpu_time_us":4888,"apply.run_wall_time_us":280534,"num_ops":18,"prepare.queue_time_us":447218,"prepare.run_cpu_time_us":36102,"prepare.run_wall_time_us":422548,"raft.queue_time_us":325907,"raft.run_cpu_time_us":15801,"raft.run_wall_time_us":169563,"replication_time_us":1216629,"thread_start_us":896035,"threads_started":4,"wal-append.queue_time_us":239149,"wal-append.run_cpu_time_us":64992,"wal-append.run_wall_time_us":2327439}]]}
I20260620 19:59:26.121752 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.122869 6905 raft_consensus.cc:2410] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:26.125217 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.121104 6345 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:26.128582 6307 consensus_meta.cc:287] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.874s user 0.004s sys 0.021s
I20260620 19:59:26.128659 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.129678 6910 raft_consensus.cc:2410] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260620 19:59:26.125592 6302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1653) took 1318 ms. Trace:
W20260620 19:59:26.074364 8808 consensus_meta.cc:287] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.514s user 0.012s sys 0.009s
I20260620 19:59:26.131011 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.123318 9236 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.132395 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.132726 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1970) took 1408 ms. Trace:
I20260620 19:59:26.132498 9236 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.133196 6306 raft_consensus.cc:2393] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 9: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:26.134728 8808 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.135092 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.135943 6906 raft_consensus.cc:2393] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 6: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:26.136689 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.136977 8808 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.074088 9222 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.137470 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.138993 9222 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:26.122548 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.133095 6224 rpcz_store.cc:276] 0620 19:59:24.724431 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.834591 (+110160us) service_pool.cc:224] Handling call
0620 19:59:24.891020 (+ 56429us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:24.891379 (+ 359us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:24.891411 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:25.076934 (+185523us) log.cc:844] Serialized 26 byte log entry
0620 19:59:26.042861 (+965927us) raft_consensus.cc:1645] Marking committed up to 5
0620 19:59:26.102526 (+ 59665us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.132334 (+ 29808us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.132458 (+ 124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:26.132498 (+ 40us) raft_consensus.cc:1685] finished
0620 19:59:26.132509 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.132638 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94685,"prepare.queue_time_us":60144,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":34,"raft.queue_time_us":186069,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":9597,"thread_start_us":938602,"threads_started":1,"wal-append.queue_time_us":915386}
I20260620 19:59:26.138092 9240 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:26.145182 6301 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:26.155463 6908 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:26.153563 6307 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:26.158943 9302 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:26.151872 9240 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:26.130205 6302 rpcz_store.cc:276] 0620 19:59:24.795420 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.795988 (+ 568us) service_pool.cc:224] Handling call
0620 19:59:24.930107 (+134119us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5800003098. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57916ee 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:25.056855 (+126748us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b5800003098. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b579178c 00007fa6b57916f6 000000000036f859 00007fa6bd8098e0 00007fa6bd8d27c4 00007fa6bd8d2831 00007fa6bd8946c0 00007fa6bd87f029 00007fa6bd87ead2 00007fa6b6df3d34
0620 19:59:25.057020 (+ 165us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:25.057280 (+ 260us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:25.315373 (+258093us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.111163 (+795790us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.111413 (+ 250us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:26.111507 (+ 94us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:26.111576 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.111671 (+ 95us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.113793 (+ 2122us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":43847,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":75,"raft.queue_time_us":8960,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":175,"spinlock_wait_cycles":76021120,"thread_start_us":966712,"threads_started":2,"wal-append.queue_time_us":788083}
W20260620 19:59:26.183746 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.188584 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.191377 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.193943 9222 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.194510 9158 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.197206 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.177217 9240 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:26.202104 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.207811 6907 raft_consensus.cc:2393] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:26.196314 9025 consensus_meta.cc:287] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.623s user 0.008s sys 0.007s
W20260620 19:59:26.220969 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 100163us
I20260620 19:59:26.221647 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1691) took 1432 ms. Trace:
I20260620 19:59:26.210134 9223 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.224771 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.235834 6302 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:26.237751 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.222448 6374 rpcz_store.cc:276] 0620 19:59:24.789047 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.944179 (+155132us) service_pool.cc:224] Handling call
0620 19:59:24.944485 (+ 306us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:24.945837 (+ 1352us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:24.945874 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:24.946516 (+ 642us) log.cc:844] Serialized 52 byte log entry
0620 19:59:25.550630 (+604114us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:25.933459 (+382829us) log.cc:844] Serialized 14 byte log entry
0620 19:59:25.979284 (+ 45825us) log.cc:844] Serialized 14 byte log entry
0620 19:59:25.982547 (+ 3263us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:25.995357 (+ 12810us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:26.159795 (+164438us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b180034fc00. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:26.221394 (+ 61599us) raft_consensus.cc:1685] finished
0620 19:59:26.221421 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.221585 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3021,"prepare.queue_time_us":360372,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":89,"raft.queue_time_us":24141,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":20054,"spinlock_wait_cycles":52042240,"thread_start_us":983346,"threads_started":2,"wal-append.queue_time_us":1064787}
W20260620 19:59:26.234776 8999 consensus_meta.cc:287] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.659s user 0.004s sys 0.013s
I20260620 19:59:26.243984 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:26.241757 9041 consensus_meta.cc:287] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.644s user 0.012s sys 0.005s
I20260620 19:59:26.229653 9223 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:26.251704 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.251200 6298 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
W20260620 19:59:26.255949 8864 consensus_meta.cc:287] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.693s user 0.016s sys 0.008s
I20260620 19:59:26.256385 8864 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.252452 6301 raft_consensus.cc:2376] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7. Re-sending same reply.
I20260620 19:59:26.262850 6909 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
W20260620 19:59:26.267560 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.266449 6902 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
I20260620 19:59:26.246656 8999 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.248657 9041 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.276379 6299 raft_consensus.cc:2393] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:26.278795 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1663) took 1146 ms. Trace:
I20260620 19:59:26.279134 6299 rpcz_store.cc:276] 0620 19:59:25.132340 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:25.528689 (+396349us) service_pool.cc:224] Handling call
0620 19:59:26.278582 (+749893us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54003255d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:26.278740 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1431086848}
I20260620 19:59:26.279830 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.280146 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.281574 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:26.284343 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.284413 6300 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:26.281770 6907 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:26.287236 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:26.287899 6299 raft_consensus.cc:2376] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7. Re-sending same reply.
W20260620 19:59:26.288268 6307 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52350 (request call id 1650) took 1502 ms (client timeout 1961 ms). Trace:
W20260620 19:59:26.288754 6307 rpcz_store.cc:269] 0620 19:59:24.786356 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.786497 (+ 141us) service_pool.cc:224] Handling call
0620 19:59:26.288043 (+1501546us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54003232d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:26.288193 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":979008768}
I20260620 19:59:26.288295 6374 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:26.284054 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.292163 9223 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.292207 6905 raft_consensus.cc:2410] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 8 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260620 19:59:26.289237 6902 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260620 19:59:26.298485 8999 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:26.299011 9314 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.108s user 0.004s sys 0.000s
W20260620 19:59:26.301993 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.300263 6911 raft_consensus.cc:2376] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7. Re-sending same reply.
I20260620 19:59:26.303117 6271 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:26.304072 6909 raft_consensus.cc:1275] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260620 19:59:26.310815 6228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1972) took 1581 ms. Trace:
I20260620 19:59:26.312135 9041 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.311192 6228 rpcz_store.cc:276] 0620 19:59:24.728972 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.887621 (+158649us) service_pool.cc:224] Handling call
0620 19:59:25.077849 (+190228us) raft_consensus.cc:1471] Updating replica for 3 ops
0620 19:59:25.291780 (+213931us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:25.291807 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0620 19:59:25.292447 (+ 640us) log.cc:844] Serialized 78 byte log entry
0620 19:59:25.978930 (+686483us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:26.102423 (+123493us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.130367 (+ 27944us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.133789 (+ 3422us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.133968 (+ 179us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.310266 (+176298us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54000b68d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:26.310404 (+ 138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:26.310453 (+ 49us) raft_consensus.cc:1685] finished
0620 19:59:26.310464 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.310662 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59931,"prepare.queue_time_us":90207,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":2398,"raft.queue_time_us":65409,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":6539,"spinlock_wait_cycles":412835584,"thread_start_us":422573,"threads_started":1,"wal-append.queue_time_us":589206}
W20260620 19:59:26.314663 9303 log.cc:927] Time spent T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.123s user 0.000s sys 0.001s
I20260620 19:59:26.315949 9236 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.317625 8864 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.319149 9025 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:26.319357 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1661) took 1459 ms. Trace:
I20260620 19:59:26.319563 9025 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:26.319854 6303 rpcz_store.cc:276] 0620 19:59:24.860391 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.958050 (+ 97659us) service_pool.cc:224] Handling call
0620 19:59:24.958380 (+ 330us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:26.002702 (+1044322us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.319014 (+316312us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001a9854. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:26.319285 (+ 271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2383219072}
I20260620 19:59:26.321416 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.321902 6271 leader_election.cc:400] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5.
I20260620 19:59:26.322163 6907 raft_consensus.cc:2393] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 9: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:26.322223 6271 leader_election.cc:403] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260620 19:59:26.323107 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.324829 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.325426 6903 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.325627 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.326246 6907 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.329805 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1694) took 1191 ms. Trace:
I20260620 19:59:26.330991 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.331631 6374 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 4.
W20260620 19:59:26.332734 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 87647.4us
I20260620 19:59:26.334211 6376 raft_consensus.cc:1275] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260620 19:59:26.332747 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.335510 6381 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:26.337121 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.337700 6908 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 7
W20260620 19:59:26.338987 9312 log.cc:927] Time spent T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.166s user 0.000s sys 0.002s
I20260620 19:59:26.339849 9296 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:26.340451 9296 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:26.340804 6877 leader_election.cc:336] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.730s (SENT)
I20260620 19:59:26.341281 6877 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:26.337136 9310 log.cc:927] Time spent T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260620 19:59:26.344311 6232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 1975) took 1487 ms. Trace:
I20260620 19:59:26.346942 9233 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5.
I20260620 19:59:26.350487 6344 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:26.332850 9158 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.018s
I20260620 19:59:26.351605 9255 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:26.352035 9255 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:26.352577 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.353391 6374 raft_consensus.cc:1240] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:26.344699 6232 rpcz_store.cc:276] 0620 19:59:24.772139 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:25.078108 (+305969us) service_pool.cc:224] Handling call
0620 19:59:25.098951 (+ 20843us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:25.126312 (+ 27361us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:25.126351 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:25.131775 (+ 5424us) log.cc:844] Serialized 52 byte log entry
0620 19:59:25.502951 (+371176us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b1800280120. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57ab607 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c8402 00007fa6b84535ab 00007fa6b84590c3 00007fa6b8459471 00007fa6b8875def
0620 19:59:26.006789 (+503838us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:26.007249 (+ 460us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.007616 (+ 367us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.242205 (+234589us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.244468 (+ 2263us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:26.247969 (+ 3501us) raft_consensus.cc:1685] finished
0620 19:59:26.247987 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.259212 (+ 11225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":203132,"prepare.queue_time_us":97536,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":14799,"raft.queue_time_us":19391,"raft.run_cpu_time_us":4240,"raft.run_wall_time_us":365433,"spinlock_wait_cycles":64029056,"thread_start_us":671251,"threads_started":2,"wal-append.queue_time_us":849342}
I20260620 19:59:26.353459 6379 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:26.354558 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.354782 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.355297 6379 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:26.355428 6378 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:26.319962 9025 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:26.354462 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:26.359151 6374 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:26.352535 9255 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:26.330304 6373 rpcz_store.cc:276] 0620 19:59:25.138291 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:25.324285 (+185994us) service_pool.cc:224] Handling call
0620 19:59:25.356185 (+ 31900us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:26.098877 (+742692us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:26.098938 (+ 61us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:26.099877 (+ 939us) log.cc:844] Serialized 52 byte log entry
0620 19:59:26.231788 (+131911us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:26.242801 (+ 11013us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.245446 (+ 2645us) log.cc:844] Serialized 14 byte log entry
0620 19:59:26.303168 (+ 57722us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b18004400e0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8d22 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:26.329099 (+ 25931us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b18004400e0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b574340a 00007fa6bee885d9 00007fa6b57cb240 00007fa6b57c5ce4 00007fa6b57c5c55 00007fa6b88b2440 00007fa6b88c4af6 00007fa6b88d5667 00007fa6b88d55e8
0620 19:59:26.329322 (+ 223us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.329444 (+ 122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:26.329499 (+ 55us) raft_consensus.cc:1685] finished
0620 19:59:26.329520 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.329736 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30765,"prepare.queue_time_us":15997,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":869,"raft.queue_time_us":97264,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":3831,"spinlock_wait_cycles":15402496,"thread_start_us":838729,"threads_started":2,"wal-append.queue_time_us":160067}
I20260620 19:59:26.365094 6379 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260620 19:59:26.365864 6907 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:26.366818 9296 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:26.370891 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 184874us
I20260620 19:59:26.377252 6903 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:26.367019 9300 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:26.389302 6908 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
W20260620 19:59:26.383288 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 58810.9us
W20260620 19:59:26.384027 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.386086 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:26.388900 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:26.394740 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.392002 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.397397 6907 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:26.403915 6376 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
W20260620 19:59:26.396665 6372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 530) took 4725 ms (client timeout 2297 ms). Trace:
W20260620 19:59:26.405439 6372 rpcz_store.cc:269] 0620 19:59:21.670897 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.672160 (+ 1263us) service_pool.cc:224] Handling call
0620 19:59:26.367377 (+4695217us) spinlock_profiling.cc:243] Waited 4.17 s on lock 0x7b54001216d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:26.396520 (+ 29143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9601158400}
I20260620 19:59:26.406579 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:26.408272 6271 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:26.413861 6379 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:26.415131 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1696) took 1269 ms. Trace:
I20260620 19:59:26.415534 6380 rpcz_store.cc:276] 0620 19:59:25.138939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:25.371385 (+232446us) service_pool.cc:224] Handling call
0620 19:59:25.371700 (+ 315us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:25.371913 (+ 213us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:25.371936 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:25.641047 (+269111us) log.cc:844] Serialized 26 byte log entry
0620 19:59:26.065888 (+424841us) raft_consensus.cc:1645] Marking committed up to 10
0620 19:59:26.065971 (+ 83us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.073566 (+ 7595us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:26.284443 (+210877us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:26.399983 (+115540us) raft_consensus.cc:1685] finished
0620 19:59:26.400023 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.407944 (+ 7921us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25414912,"thread_start_us":417330,"threads_started":1,"wal-append.queue_time_us":549427}
W20260620 19:59:26.417542 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 76774us
W20260620 19:59:26.437170 6877 leader_election.cc:336] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.976s (SENT)
I20260620 19:59:26.438684 6877 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.439136 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:26.455231 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.470978 6381 raft_consensus.cc:2410] T 92212144534e4394844003016ba67a47 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260620 19:59:26.471467 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.473346 6875 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:26.482268 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 74108.7us
I20260620 19:59:26.483683 6374 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
W20260620 19:59:26.492983 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 46941.9us
I20260620 19:59:26.498692 6372 raft_consensus.cc:1275] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260620 19:59:26.498351 9233 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:26.502628 9298 consensus_queue.cc:1059] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.024s
I20260620 19:59:26.493861 9296 raft_consensus.cc:2764] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260620 19:59:26.503367 9233 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:26.503978 9298 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:26.504396 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 535) took 4739 ms. Trace:
I20260620 19:59:26.504559 6380 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:26.505332 9235 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:26.505764 9233 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.506501 9319 consensus_queue.cc:1059] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260620 19:59:26.504882 6375 rpcz_store.cc:276] 0620 19:59:21.765072 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:21.765267 (+ 195us) service_pool.cc:224] Handling call
0620 19:59:26.402918 (+4637651us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:26.439991 (+ 37073us) raft_consensus.cc:1534] Early marking committed up to index 6
0620 19:59:26.440034 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:26.440083 (+ 49us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:26.440128 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:26.444734 (+ 4606us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:26.504071 (+ 59337us) spinlock_profiling.cc:243] Waited 4.62 s on lock 0x7b54001216d0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:26.504212 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25201,"raft.queue_time_us":31629,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7002,"spinlock_wait_cycles":10614529920}
I20260620 19:59:26.504374 9298 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
W20260620 19:59:26.509114 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 71990.6us
I20260620 19:59:26.509357 9298 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.10, Last appended by leader: 10, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.511080 9298 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:26.514715 6345 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:26.516449 6345 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:26.517127 9158 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:26.522533 9294 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:26.523156 9294 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.525318 9294 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.490238 9309 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.525990 9309 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.528438 9309 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.490939 8864 consensus_queue.cc:1059] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260620 19:59:26.529771 8864 raft_consensus.cc:3055] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:26.530144 8864 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:26.531009 8864 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.532203 8864 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:26.536871 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.507709 9319 raft_consensus.cc:3055] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:26.549117 9319 raft_consensus.cc:740] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:26.444975 9302 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260620 19:59:26.518342 6345 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:26.565089 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:26.565935 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 63935.5us
I20260620 19:59:26.565900 6375 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.564314 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.569734 9308 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:26.567276 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:26.566768 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 66506.6us
I20260620 19:59:26.498479 9240 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:26.563453 9300 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:26.597455 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 79168.8us
I20260620 19:59:26.497711 9025 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.608999 9300 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.609961 6379 raft_consensus.cc:2468] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:26.611287 9300 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:26.612107 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 109996us
I20260620 19:59:26.613199 6271 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:26.613773 6375 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:26.614739 6195 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:26.615608 9133 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:26.616217 9133 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:26.617123 6195 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.617000 9255 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.618335 9133 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.619241 9255 leader_election.cc:290] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.620939 9240 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.626730 9135 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
W20260620 19:59:26.629812 9325 log.cc:927] Time spent T 00000000000000000000000000000000 P 8bc9d9c8cc8a4f5d911f3ec139791862: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260620 19:59:26.632239 6874 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:26.633033 9296 consensus_queue.cc:1059] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:59:26.633445 6874 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:26.634269 9296 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:26.596014 9223 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:26.634904 9296 raft_consensus.cc:3055] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:26.635179 9223 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:26.635285 9296 raft_consensus.cc:740] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:26.636222 9296 consensus_queue.cc:260] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:26.637498 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.637416 9296 raft_consensus.cc:3060] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:26.647938 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 46745.5us
I20260620 19:59:26.559748 9319 consensus_queue.cc:260] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.646915 9005 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:26.658272 6345 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:26.655334 6874 leader_election.cc:304] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:26.648360 9262 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:26.652436 9170 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:26.657100 9005 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:26.656454 9240 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:26.675002 9300 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:26.675452 9300 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:26.675840 9300 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:26.655476 9294 consensus_queue.cc:1059] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260620 19:59:26.691345 9319 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:26.673545 9005 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:26.695922 9025 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.696849 9294 raft_consensus.cc:3055] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:26.697294 9294 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:26.673509 9302 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:26.698271 9294 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:26.703675 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.699524 6273 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
W20260620 19:59:26.720499 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 62306.4us
W20260620 19:59:26.727384 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 171207us
I20260620 19:59:26.728945 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.729921 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:26.730224 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 76676.3us
I20260620 19:59:26.732999 6273 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.730494 6228 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:26.737097 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.691562 9238 consensus_queue.cc:1059] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260620 19:59:26.746268 9224 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:59:26.750907 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.729506 9294 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:26.756918 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 87533.8us
I20260620 19:59:26.739943 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:26.766722 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.767030 9128 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:26.767657 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.740383 9025 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:26.770565 6227 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:26.775775 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.775938 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.776948 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.777141 6223 raft_consensus.cc:1240] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260620 19:59:26.793880 6232 raft_consensus.cc:2393] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 9: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:26.795882 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.779417 9300 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.783124 9296 raft_consensus.cc:3055] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:26.783385 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.729688 9041 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260620 19:59:26.764144 6345 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:26.801553 6907 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:26.798485 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.798949 6271 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:26.800410 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 72362.6us
I20260620 19:59:26.802153 6228 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:26.786015 9240 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:26.806077 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:26.804035 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:26.806490 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 75205.7us
I20260620 19:59:26.804322 9223 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:26.807375 6273 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.807575 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:26.799556 9296 raft_consensus.cc:740] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:26.808457 6228 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:26.808617 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.809190 6273 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:26.809082 9296 consensus_queue.cc:260] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.809998 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.810146 6273 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:26.785095 9005 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.793460 9025 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:26.807646 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:26.828893 9328 log.cc:927] Time spent T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20260620 19:59:26.831566 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:26.826416 9235 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260620 19:59:26.841928 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.842108 9235 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:26.815212 6196 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:26.831171 6875 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:26.813577 9302 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:26.816177 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.843618 6229 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:26.843740 6196 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.844522 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:26.843766 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.844666 6875 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:26.813750 6231 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:26.810443 9296 raft_consensus.cc:3060] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:26.845638 9338 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:26.845788 9319 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:26.846074 9338 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:26.846540 9338 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:26.831355 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:26.840386 9025 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.847416 6372 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 7
I20260620 19:59:26.811259 6908 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:26.848289 9238 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.818069 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.848832 9238 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:26.810846 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:26.811064 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 69941.9us
I20260620 19:59:26.846477 6231 raft_consensus.cc:1275] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260620 19:59:26.832070 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:26.811075 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:26.839959 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:26.817185 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:26.845170 6196 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:26.852609 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.845113 6229 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:26.831759 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:26.849193 6224 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:26.854307 9128 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:26.814527 9223 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:26.857395 9319 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:26.857827 9319 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:26.858175 9319 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:26.857395 9223 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:26.812960 9233 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:26.858829 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:26.859617 6232 raft_consensus.cc:2393] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 9: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:26.858865 9223 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:26.850971 6381 raft_consensus.cc:3060] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.855324 6377 raft_consensus.cc:3060] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:26.855530 6226 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:26.827750 9240 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:26.865222 6373 raft_consensus.cc:1275] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:26.865137 6273 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:26.864466 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 70001.9us
I20260620 19:59:26.842579 9235 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:26.866629 6198 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:26.870379 6230 raft_consensus.cc:1275] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260620 19:59:26.821017 9262 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:26.872359 6875 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:26.858362 9005 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.876012 9128 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260620 19:59:26.876369 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.875284 9300 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.871984 6345 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:26.885320 6372 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:26.870092 9236 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:26.889798 9236 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:26.889922 9240 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:26.890231 9236 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:26.891032 9114 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:26.891497 9114 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:26.891912 9114 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:26.865748 9233 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.021s
I20260620 19:59:26.899389 6345 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:26.899623 9236 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.900259 9170 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260620 19:59:26.901027 9170 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:26.901412 9170 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:26.901783 9170 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:26.902172 9236 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.903647 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:26.905601 9319 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.886816 9128 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:26.907161 9128 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:26.910796 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.912648 6196 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.913451 9133 consensus_queue.cc:1059] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260620 19:59:26.914973 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 48442.3us
I20260620 19:59:26.914319 9133 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:26.918550 6198 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:26.920598 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 55565.4us
I20260620 19:59:26.920625 6228 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:26.914343 9135 consensus_queue.cc:1048] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260620 19:59:26.921756 8864 raft_consensus.cc:2749] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:26.918354 9240 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:26.924686 9240 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:26.925231 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:26.925689 6273 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.922001 8829 consensus_queue.cc:1048] T f202638d921b4ef2b75223ca54d70e04 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.007s
I20260620 19:59:26.928234 9236 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:26.936775 9319 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.936945 9224 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 19:59:26.937731 9319 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:26.938179 9319 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:26.938822 9339 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.939417 9339 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.925501 9240 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.952987 6227 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:26.955395 6381 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:26.956938 6377 raft_consensus.cc:2468] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:26.961867 9294 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:26.944618 9235 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.962458 9294 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.963932 9235 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:26.949222 9319 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.965085 9294 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:26.957858 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 88222.3us
I20260620 19:59:26.948127 9238 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.987054 9025 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:26.989329 9025 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.958405 9302 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260620 19:59:26.994349 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:27.014714 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 111144us
I20260620 19:59:26.949632 9133 raft_consensus.cc:697] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:27.023653 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 89303.9us
I20260620 19:59:26.995380 9128 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:26.996982 9170 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:27.002754 9240 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:26.996982 9338 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:27.012252 6877 leader_election.cc:336] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.856s (SENT)
I20260620 19:59:27.027757 6877 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:26.942711 9339 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:26.994716 9223 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:27.025336 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 111213us
I20260620 19:59:27.028245 6345 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.039439 6198 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:27.047252 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:27.052418 6198 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:27.055415 6271 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.058053 6196 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.060524 9223 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:27.060679 9338 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:27.060328 9133 consensus_queue.cc:237] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.062288 9254 raft_consensus.cc:2804] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:27.062556 8829 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:27.063215 9341 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260620 19:59:27.063164 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 50961.8us
I20260620 19:59:27.050315 9262 consensus_queue.cc:1048] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.014s
I20260620 19:59:27.051954 9339 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:27.052824 9302 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260620 19:59:27.069612 6196 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.027324 9114 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:27.068754 9135 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
W20260620 19:59:27.077123 6130 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47082 (request call id 408) took 11868 ms (client timeout 15000 ms). Trace:
I20260620 19:59:27.076397 6196 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.072906 8829 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:27.088580 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:27.081005 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 155399us
I20260620 19:59:27.084923 9344 raft_consensus.cc:2804] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:27.083488 8829 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:27.078612 9342 raft_consensus.cc:2804] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:27.082835 9128 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:27.077095 9319 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:27.104278 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.104804 6902 raft_consensus.cc:1275] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:27.107044 6304 raft_consensus.cc:3060] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:27.104948 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.105327 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:27.107859 6911 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:27.105504 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:27.105865 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:27.119252 6273 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:27.122279 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 63343.5us
I20260620 19:59:27.128927 6271 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:27.129245 6303 raft_consensus.cc:2410] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260620 19:59:27.130077 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:27.132539 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 48314.1us
W20260620 19:59:27.079347 6130 rpcz_store.cc:269] 0620 19:59:15.209142 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:15.211993 (+ 2851us) service_pool.cc:224] Handling call
0620 19:59:15.300031 (+ 88038us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b54001a0590. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b57918b2 00007fa6be70f659 00007fa6be6a78b6 00007fa6be678968 00007fa6be7770ae 00007fa6bcc3c4a4 00007fa6bcc3c432 00007fa6bcc3c361 00007fa6bcc3c2dc
0620 19:59:27.077055 (+11777024us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:59:26.231762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:59:26.254786 (+ 23024us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:59:26.254839 (+ 53us) write_op.cc:435] Acquired schema lock
0620 19:59:26.254855 (+ 16us) tablet.cc:662] Decoding operations
0620 19:59:26.255529 (+ 674us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:59:26.255627 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0620 19:59:26.255652 (+ 25us) tablet.cc:685] Acquiring locks for 4 operations
0620 19:59:26.256128 (+ 476us) tablet.cc:701] Row locks acquired
0620 19:59:26.256148 (+ 20us) write_op.cc:260] PREPARE: finished
0620 19:59:26.256316 (+ 168us) write_op.cc:270] Start()
0620 19:59:26.256469 (+ 153us) write_op.cc:276] Timestamp: P: 1781985566256257 usec, L: 0
0620 19:59:26.256487 (+ 18us) op_driver.cc:348] REPLICATION: starting
0620 19:59:26.275280 (+ 18793us) log.cc:844] Serialized 1691 byte log entry
0620 19:59:26.744149 (+468869us) op_driver.cc:464] REPLICATION: finished
0620 19:59:26.890187 (+146038us) write_op.cc:301] APPLY: starting
0620 19:59:26.890286 (+ 99us) tablet.cc:1367] starting BulkCheckPresence
0620 19:59:26.890465 (+ 179us) tablet.cc:1370] finished BulkCheckPresence
0620 19:59:26.890628 (+ 163us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:59:26.891539 (+ 911us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:59:26.891858 (+ 319us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0620 19:59:26.891903 (+ 45us) write_op.cc:312] APPLY: finished
0620 19:59:26.892206 (+ 303us) log.cc:844] Serialized 41 byte log entry
0620 19:59:26.895818 (+ 3612us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:26.895915 (+ 97us) write_op.cc:454] Released schema lock
0620 19:59:26.896092 (+ 177us) write_op.cc:341] FINISH: Updating metrics
0620 19:59:26.896506 (+ 414us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:26.896538 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1849513,"reported_tablets":34,"spinlock_wait_cycles":56871168,"child_traces":[["op",{"apply.queue_time_us":145731,"apply.run_cpu_time_us":2668,"apply.run_wall_time_us":6307,"num_ops":4,"prepare.queue_time_us":377705,"prepare.run_cpu_time_us":18470,"prepare.run_wall_time_us":192276,"raft.queue_time_us":109267,"raft.run_cpu_time_us":30402,"raft.run_wall_time_us":125022,"replication_time_us":487468,"thread_start_us":630230,"threads_started":4,"wal-append.queue_time_us":255590}]]}
I20260620 19:59:27.135869 6271 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:27.128144 9223 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:27.142263 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:27.135025 9170 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:27.132361 9254 raft_consensus.cc:697] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:27.137856 9235 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:27.153277 6273 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.156116 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:27.146728 9223 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:27.157490 6198 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.150840 9114 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:27.147419 9135 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:27.142854 9342 raft_consensus.cc:697] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:27.149295 9302 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:27.148944 9344 raft_consensus.cc:697] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:27.161240 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.167937 9254 consensus_queue.cc:237] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.168385 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.169611 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.170290 6231 raft_consensus.cc:2376] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7. Re-sending same reply.
I20260620 19:59:27.154258 9224 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:27.170847 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.171340 9224 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:27.171980 9302 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.172293 9224 consensus_queue.cc:237] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.174132 9302 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:27.174844 6345 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.175392 9350 raft_consensus.cc:2804] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:27.175979 9350 raft_consensus.cc:697] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:27.154731 9235 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:27.170920 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.176021 9170 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:27.176657 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.184077 6223 raft_consensus.cc:2393] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:27.188227 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.160166 9223 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:27.142066 9025 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260620 19:59:27.186457 6877 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.271s (SENT) [suppressed 44 similar messages]
I20260620 19:59:27.193677 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.161501 9236 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:27.195086 6228 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:27.195422 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.198469 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.184067 9235 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:27.192966 9341 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:27.184221 9302 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260620 19:59:27.210311 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.173130 9348 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20260620 19:59:27.212976 6345 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.217130 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.194463 9114 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260620 19:59:27.226037 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:27.226971 6910 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
W20260620 19:59:27.220011 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 70454.2us
I20260620 19:59:27.215013 9302 consensus_queue.cc:1243] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260620 19:59:27.212287 9341 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:27.236472 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.246260 6273 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:27.246225 9356 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:27.249658 9360 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:27.250754 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:27.254276 9319 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:27.219038 6128 catalog_manager.cc:5696] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 4 to 6, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 6 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } } }
I20260620 19:59:27.256247 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.256975 6228 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:27.259619 9343 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:27.260098 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:27.260725 6377 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
W20260620 19:59:27.261598 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 53393us
I20260620 19:59:27.262291 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.205080 9342 consensus_queue.cc:237] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:27.263952 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 77507.4us
I20260620 19:59:27.269377 6875 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:27.320235 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 165121us
I20260620 19:59:27.324293 6271 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:27.325593 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:27.331429 6877 leader_election.cc:336] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 3.062s (SENT)
I20260620 19:59:27.244697 9341 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:27.332834 6271 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:27.335597 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 73361.9us
I20260620 19:59:27.337776 6877 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:27.338480 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 69161.1us
W20260620 19:59:27.345016 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 98822.6us
I20260620 19:59:27.331696 6128 catalog_manager.cc:5696] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 4 to 5, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 5 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.260107 9343 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:27.350637 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.323411 9365 consensus_queue.cc:1048] T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:27.243045 8864 consensus_queue.cc:1048] T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.107s
I20260620 19:59:27.331949 9302 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:27.359684 6128 catalog_manager.cc:5696] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 2 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:27.208073 9350 consensus_queue.cc:237] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:27.367835 9360 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260620 19:59:27.357834 9339 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:27.357686 9338 raft_consensus.cc:2804] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
W20260620 19:59:27.376950 6198 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:27.359243 9343 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:27.385546 6198 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.389703 6223 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260620 19:59:27.390094 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:27.390681 6379 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:27.390897 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.237298 9135 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:27.395218 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 63799.4us
I20260620 19:59:27.375398 9356 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:27.414894 6229 raft_consensus.cc:1240] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:27.415056 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:27.415376 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.428300 6230 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:27.416043 6345 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.416270 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.418857 6128 catalog_manager.cc:5696] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 7, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 7 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.255069 9344 consensus_queue.cc:237] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.430585 6128 catalog_manager.cc:5696] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:27.432415 6128 catalog_manager.cc:5696] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 5 to 6, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 6 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.434460 6128 catalog_manager.cc:5696] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 4 to 5, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:59:27.435719 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:27.436532 6128 catalog_manager.cc:5696] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:27.438256 6128 catalog_manager.cc:5696] T db666bffcf904fa59d91db75dc91c8b0 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 6, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 6 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.440479 6128 catalog_manager.cc:5696] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 2 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:27.446286 6379 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:27.393272 9338 raft_consensus.cc:697] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:59:27.446715 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:27.409165 9365 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260620 19:59:27.455623 6195 leader_election.cc:336] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.741s (SENT)
I20260620 19:59:27.456017 6195 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:27.442415 6128 catalog_manager.cc:5696] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:27.444444 9356 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:27.446669 9351 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:27.452078 9025 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:27.446197 9354 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:27.452122 9365 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:27.472859 6128 catalog_manager.cc:5696] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 6 to 7, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 7 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:59:27.467140 9346 log.cc:927] Time spent T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.254s user 0.001s sys 0.001s
W20260620 19:59:27.479588 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 146793us
W20260620 19:59:27.476667 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 130850us
I20260620 19:59:27.482805 6273 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.487323 6911 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:27.469283 9360 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:27.472594 9356 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:27.490810 6128 catalog_manager.cc:5696] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 6 to 7, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 7 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.504144 9367 consensus_queue.cc:1048] T 107df92b2ae944819f7fbc4cb216d56f P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:27.505061 9375 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:27.505548 9367 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:27.505963 9367 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:27.506321 9367 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:27.476276 9025 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:27.475471 9351 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:27.479385 9365 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:27.509397 6273 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:27.511024 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 59457us
I20260620 19:59:27.476040 9354 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:27.511787 6195 leader_election.cc:336] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.780s (SENT)
W20260620 19:59:27.513765 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 135263us
I20260620 19:59:27.513967 6195 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:27.522087 6304 raft_consensus.cc:2468] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:27.524850 6128 catalog_manager.cc:5696] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 4 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.527501 6345 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.510736 9025 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:27.513530 9351 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:27.516441 9354 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:27.536360 6271 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:27.541478 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:27.551434 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 50826.9us
I20260620 19:59:27.543742 9338 consensus_queue.cc:237] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:27.552829 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 78487.1us
I20260620 19:59:27.543116 9236 raft_consensus.cc:697] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:27.555449 9236 consensus_queue.cc:237] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.558722 6128 catalog_manager.cc:5696] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 4 to 5, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 5 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.537940 9360 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:27.553812 6198 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.543257 9224 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:27.566936 6128 catalog_manager.cc:5696] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:59:27.575795 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 57261.4us
I20260620 19:59:27.575505 6128 catalog_manager.cc:5696] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 5 to 7, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 7 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.568854 9224 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:27.586486 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 45069.4us
I20260620 19:59:27.599226 6128 catalog_manager.cc:5696] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 5 to 6, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 6 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:27.564428 9223 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.611909 6345 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:27.616230 6195 leader_election.cc:336] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.961s (SENT)
I20260620 19:59:27.619905 6128 catalog_manager.cc:5696] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 7, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 7 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.602365 9224 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:27.635594 6128 catalog_manager.cc:5696] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 2 to 5, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to 39662d01be7a44f682d160227f457661 (127.4.244.130). New cstate: current_term: 5 leader_uuid: "39662d01be7a44f682d160227f457661" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
W20260620 19:59:27.640707 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 72188.5us
I20260620 19:59:27.623646 9360 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:27.642835 6273 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:27.587517 9369 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260620 19:59:27.651736 6226 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:27.646319 6128 catalog_manager.cc:5696] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 4 to 7, leader changed from eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 7 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260620 19:59:27.651068 9371 log.cc:927] Time spent T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
W20260620 19:59:27.656698 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 61835.6us
W20260620 19:59:27.661065 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 44873.4us
I20260620 19:59:27.660331 6128 catalog_manager.cc:5696] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 3 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 4 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.662071 6345 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.668812 6907 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:27.671559 6875 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.679167 6128 catalog_manager.cc:5696] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 reported cstate change: term changed from 1 to 5, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132). New cstate: current_term: 5 leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260620 19:59:27.685539 9353 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:27.685817 9223 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:27.686177 9353 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:27.680099 9360 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:27.706514 9381 consensus_queue.cc:1048] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
W20260620 19:59:27.708089 9370 log.cc:927] Time spent T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
W20260620 19:59:27.708799 9374 log.cc:927] Time spent T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260620 19:59:27.709249 9382 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
W20260620 19:59:27.702898 9363 log.cc:927] Time spent T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
I20260620 19:59:27.711076 9367 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:27.712630 9372 log.cc:927] Time spent T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260620 19:59:27.713413 9367 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:27.713830 9383 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260620 19:59:27.714540 9383 consensus_queue.cc:1243] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260620 19:59:27.686872 9379 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:27.715251 9379 raft_consensus.cc:697] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:27.716140 9379 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.717185 6376 raft_consensus.cc:1275] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:27.717092 9223 raft_consensus.cc:3055] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:27.717725 9223 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:27.718634 9223 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:27.719836 9223 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:27.721519 9382 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:27.722263 9368 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20260620 19:59:27.723052 9382 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.687680 9128 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:27.727051 9128 raft_consensus.cc:697] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:59:27.734496 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 71089.6us
W20260620 19:59:27.740716 6195 leader_election.cc:336] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.130s (SENT)
I20260620 19:59:27.737284 6230 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:27.732971 9367 raft_consensus.cc:2804] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
W20260620 19:59:27.737057 9235 consensus_meta.cc:287] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.524s user 0.009s sys 0.002s
W20260620 19:59:27.778893 6195 leader_election.cc:336] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.228s (SENT)
W20260620 19:59:27.781232 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 141390us
I20260620 19:59:27.784402 6195 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:27.787388 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.789181 6345 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.745395 9114 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260620 19:59:27.781898 9233 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.006s sys 0.006s
I20260620 19:59:27.727926 9128 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.793265 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.801302 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.796474 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:27.802006 6198 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:27.803869 6301 raft_consensus.cc:2376] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5. Re-sending same reply.
I20260620 19:59:27.796898 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.810680 6300 raft_consensus.cc:2368] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8.
I20260620 19:59:27.812052 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:27.801503 9233 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.007s sys 0.007s
I20260620 19:59:27.828965 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.830384 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.832335 6345 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.807134 9380 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:27.813002 9170 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:27.837996 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.807243 9367 raft_consensus.cc:697] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:27.836216 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 178296us
I20260620 19:59:27.786278 9235 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.802687 9025 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:27.849334 6271 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:27.833949 9360 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:27.812300 9353 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:27.869020 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.869074 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.890305 6344 leader_election.cc:304] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:27.864734 6345 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.883854 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.893826 6306 raft_consensus.cc:3060] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:27.864511 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.872287 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.902060 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:27.903226 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:27.845592 9294 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:27.897756 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:27.882102 6304 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:27.895347 6298 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260620 19:59:27.942327 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:27.894384 6307 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:27.939622 6231 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
W20260620 19:59:27.872287 9373 log.cc:927] Time spent T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.285s user 0.004s sys 0.000s
I20260620 19:59:27.940791 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:27.944186 6231 pending_rounds.cc:85] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 7
W20260620 19:59:27.947834 6877 leader_election.cc:336] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.527s (SENT)
I20260620 19:59:27.962534 6305 raft_consensus.cc:1275] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260620 19:59:27.961326 6303 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:27.965586 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:27.965782 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 101116us
I20260620 19:59:27.901285 9170 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:27.966531 6345 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.968240 6909 raft_consensus.cc:1275] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260620 19:59:27.961397 6231 pending_rounds.cc:107] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260620 19:59:27.970749 6231 raft_consensus.cc:2889] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:27.973251 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:27.960131 8829 consensus_meta.cc:287] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.868s user 0.012s sys 0.008s
I20260620 19:59:27.986338 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.863639 9233 consensus_queue.cc:1059] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260620 19:59:27.987386 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:27.987664 6303 raft_consensus.cc:1240] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:27.994979 6874 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.868875 9388 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.172s
I20260620 19:59:28.002031 6302 raft_consensus.cc:1240] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:27.975641 9360 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:27.988054 8829 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:28.019802 9341 consensus_meta.cc:287] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.661s user 0.009s sys 0.001s
I20260620 19:59:28.021574 8829 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.022979 6232 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 8: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:27.972366 9294 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.025699 9235 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.025851 9294 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:27.997437 6305 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260620 19:59:28.026901 9025 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:28.039211 9365 consensus_meta.cc:287] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.515s user 0.004s sys 0.006s
I20260620 19:59:28.039682 9365 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:28.040885 6300 raft_consensus.cc:2368] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
W20260620 19:59:28.006763 9389 log.cc:927] Time spent T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260620 19:59:28.006685 9005 raft_consensus.cc:2764] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260620 19:59:28.007176 9390 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:28.008927 9358 raft_consensus.cc:2804] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election won for term 7
W20260620 19:59:28.011057 9398 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20260620 19:59:28.016923 9343 consensus_meta.cc:287] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.602s user 0.006s sys 0.002s
I20260620 19:59:28.044281 6307 raft_consensus.cc:2410] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260620 19:59:28.068789 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:27.986042 9367 consensus_queue.cc:237] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.020210 9341 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:28.091755 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.080885 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260620 19:59:28.092589 6377 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:28.098958 9360 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:28.099761 9358 raft_consensus.cc:697] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:28.100735 9358 consensus_queue.cc:237] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:28.102423 6877 leader_election.cc:336] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.682s (SENT)
I20260620 19:59:28.093417 9390 raft_consensus.cc:697] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:28.102902 6877 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.103883 9005 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:28.103796 9390 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:28.101251 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.120890 9365 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:28.123790 9224 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:28.125156 6301 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:28.129483 6375 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:28.132997 6306 raft_consensus.cc:2468] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
W20260620 19:59:28.138813 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 143897us
W20260620 19:59:28.139947 6874 leader_election.cc:336] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 2.126s (SENT)
I20260620 19:59:28.140137 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.140442 6874 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.138211 6344 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:28.141721 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.142374 6372 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:28.143142 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.143716 6380 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:28.144549 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.148854 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.149405 6381 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:28.150074 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:28.150650 6226 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:28.150831 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 82124.9us
I20260620 19:59:28.153450 6231 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:28.155565 6373 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [4.3-8.4]
I20260620 19:59:28.156487 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.156971 6373 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:28.157871 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.163353 6381 raft_consensus.cc:2468] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
W20260620 19:59:28.161218 9356 consensus_meta.cc:287] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.639s user 0.010s sys 0.000s
I20260620 19:59:28.170225 9356 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.173224 6299 raft_consensus.cc:2393] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:28.173202 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:28.176082 6373 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:28.162786 6195 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:28.178864 6226 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:28.187798 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.195712 6195 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:28.196524 6345 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.148630 9406 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.154428 9302 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:28.094209 9343 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.202797 6195 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:28.211159 6195 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:28.212200 6195 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
W20260620 19:59:28.212301 6877 leader_election.cc:336] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.064s (SENT)
I20260620 19:59:28.205643 6372 raft_consensus.cc:2468] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
W20260620 19:59:28.206140 9338 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.007s sys 0.000s
I20260620 19:59:28.187885 9342 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260620 19:59:28.158237 9238 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:28.202257 6381 raft_consensus.cc:3055] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
W20260620 19:59:28.187754 9351 consensus_meta.cc:287] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.644s user 0.005s sys 0.005s
I20260620 19:59:28.222185 6381 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:28.157554 9005 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.218556 6375 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260620 19:59:28.219802 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:28.220826 6306 raft_consensus.cc:1240] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:28.223228 6381 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.204880 9388 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260620 19:59:28.164247 9233 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260620 19:59:28.232156 6381 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:28.233101 6302 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:28.243032 6298 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:28.243520 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:28.245515 6305 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:28.246304 9347 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:28.246750 9347 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:28.247156 9347 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:28.208143 9302 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:28.250344 9302 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:28.255364 9262 consensus_queue.cc:1048] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:28.257668 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.258270 6372 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:28.259972 6195 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:28.261914 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:28.206950 9413 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260620 19:59:28.218036 9338 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.007s sys 0.000s
I20260620 19:59:28.273630 6273 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.275574 9352 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:28.275758 9356 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.279279 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:28.280017 9238 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:28.315871 6877 leader_election.cc:336] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.106s (SENT)
I20260620 19:59:28.315699 9238 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.316350 6877 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:28.316612 9375 consensus_queue.cc:1059] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:59:28.317799 9041 raft_consensus.cc:3055] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:28.318192 9041 raft_consensus.cc:740] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:28.319145 9041 consensus_queue.cc:260] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:28.232333 9025 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:28.284317 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:28.285837 6381 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
W20260620 19:59:28.287019 9354 consensus_meta.cc:287] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.743s user 0.005s sys 0.005s
I20260620 19:59:28.265111 9262 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:28.288139 9224 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:28.217818 9240 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:28.261253 9005 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.289111 9343 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.286970 9412 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.290289 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:28.347479 6303 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 3.
I20260620 19:59:28.347713 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:28.298257 6372 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:28.305281 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:28.286883 9341 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:28.313875 9302 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.317821 9238 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:28.314729 9413 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:28.318079 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.213177 9406 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:28.324692 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 55502.3us
I20260620 19:59:28.419039 9406 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:28.338016 9223 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260620 19:59:28.420958 6195 leader_election.cc:336] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141): Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.514s (SENT)
I20260620 19:59:28.421466 6195 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:28.223716 9294 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260620 19:59:28.243309 9369 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.327378 6381 pending_rounds.cc:85] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 7
I20260620 19:59:28.336750 6307 raft_consensus.cc:2393] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:28.423929 6381 pending_rounds.cc:107] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260620 19:59:28.424247 6381 raft_consensus.cc:2889] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:28.423951 9369 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.427026 6305 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 8: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:59:28.428143 9398 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260620 19:59:28.428673 9413 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.342685 6380 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:28.336508 9025 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.243500 9351 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:28.348253 6301 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:28.333925 9338 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:28.332697 9133 consensus_queue.cc:1048] T 1e39bd0f2de444d2acf683f9f18716da P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260620 19:59:28.409912 6375 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:28.354534 6877 leader_election.cc:336] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.203s (SENT)
I20260620 19:59:28.410508 6306 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
W20260620 19:59:28.413081 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 216640us
I20260620 19:59:28.413439 6198 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:28.415417 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.416182 9005 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:28.416379 6300 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:28.416587 9240 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.450699 9352 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:28.409189 9412 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.451153 9240 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.452025 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:28.452775 6908 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 6: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:28.453001 9412 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.453343 9240 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.453652 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:28.416771 9376 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.437281 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:28.434556 6271 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:28.447736 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 174159us
I20260620 19:59:28.451457 9294 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.348529 6298 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
W20260620 19:59:28.456393 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.426488 9369 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.458061 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:28.409582 9339 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.456053 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:28.346935 9114 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.132s
I20260620 19:59:28.337751 9354 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:28.524777 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 170260us
I20260620 19:59:28.640604 6273 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.461910 6304 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:28.515575 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:28.522478 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 87980.2us
W20260620 19:59:28.487996 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 74623.4us
I20260620 19:59:28.524364 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:28.609442 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 64084.9us
I20260620 19:59:28.655164 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:28.498760 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.657666 6904 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:28.654292 6273 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:28.523525 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:28.526523 9302 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.659881 6375 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
W20260620 19:59:28.642525 6877 leader_election.cc:336] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.876s (SENT)
W20260620 19:59:28.608212 9393 log.cc:927] Time spent T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.648s user 0.000s sys 0.003s
I20260620 19:59:28.452035 9005 raft_consensus.cc:3055] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:28.660166 6225 raft_consensus.cc:2393] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:28.681505 6303 raft_consensus.cc:1240] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:28.515199 9262 raft_consensus.cc:2804] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:28.687057 6877 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.448361 9388 consensus_queue.cc:1059] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.226s
I20260620 19:59:28.660715 9224 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:28.689699 6273 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:28.443370 9238 consensus_queue.cc:1059] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.226s
I20260620 19:59:28.703681 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:28.658744 9390 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:28.703863 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.458343 9254 consensus_queue.cc:1059] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.228s
I20260620 19:59:28.705722 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:28.707720 6271 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.608894 9413 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.596057 9351 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:28.708228 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 65743.3us
I20260620 19:59:28.708734 6273 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.658582 9424 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:28.715431 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:28.713436 6877 leader_election.cc:336] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.425s (SENT)
I20260620 19:59:28.709352 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:28.716735 6877 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:28.717159 6300 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:28.719631 6227 raft_consensus.cc:1240] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:28.714835 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:28.720711 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:28.689700 9005 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:28.726688 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:28.669389 9319 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.046s
I20260620 19:59:28.730923 6224 raft_consensus.cc:2393] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:28.658021 9339 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.660555 9294 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.747273 9387 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:28.727231 8864 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260620 19:59:28.748270 9387 raft_consensus.cc:697] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:28.749434 9294 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:28.749208 9387 consensus_queue.cc:237] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.749938 9406 raft_consensus.cc:3055] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:28.750321 9406 raft_consensus.cc:740] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:28.747416 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:28.750782 9339 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:28.751245 6904 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:28.753393 9420 raft_consensus.cc:2804] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:28.753925 9238 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:28.754001 9420 raft_consensus.cc:697] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:28.755241 9420 consensus_queue.cc:237] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.752290 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.710058 9025 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.752159 9354 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.752979 9400 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260620 19:59:28.759533 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.707154 9262 raft_consensus.cc:697] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:28.769179 9418 log.cc:927] Time spent T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260620 19:59:28.775017 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.776023 6379 raft_consensus.cc:2368] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 8. Current term is 9.
I20260620 19:59:28.706307 9392 consensus_queue.cc:1048] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
W20260620 19:59:28.790380 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.700850 9347 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:28.795476 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 86763us
I20260620 19:59:28.739531 9424 raft_consensus.cc:697] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:28.798820 9005 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.799669 9367 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:28.799983 9005 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:28.800940 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.800843 9424 consensus_queue.cc:237] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:28.808760 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.810401 6379 raft_consensus.cc:2410] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:28.718801 9390 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.487596 9376 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.812793 9390 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:28.791988 9338 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:28.821549 6273 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:28.829453 6273 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:28.830993 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.832630 6273 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:28.830441 9392 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:28.836957 9392 raft_consensus.cc:697] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:28.837778 9392 consensus_queue.cc:237] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.751255 9406 consensus_queue.cc:260] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.840562 6228 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:28.846431 6907 raft_consensus.cc:1275] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260620 19:59:28.850703 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 137279us
W20260620 19:59:28.851253 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:28.860872 6910 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260620 19:59:28.861147 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:28.861569 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:28.861761 6907 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:28.863616 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:28.864199 6911 raft_consensus.cc:3060] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:28.828543 9367 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260620 19:59:28.872071 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.873979 6380 raft_consensus.cc:1275] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260620 19:59:28.848855 9302 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.916958 6377 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260620 19:59:28.921996 6300 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 4.
I20260620 19:59:28.924731 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:28.937182 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 104637us
I20260620 19:59:28.928624 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:28.937537 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.882875 6227 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260620 19:59:28.885418 6379 raft_consensus.cc:2393] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 9: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:28.942795 6907 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:28.958817 6373 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:28.879884 6908 raft_consensus.cc:1275] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:28.970088 6228 raft_consensus.cc:1240] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:28.972929 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.827900 9338 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:28.870890 9233 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:28.875972 9238 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:28.991514 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:28.991545 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.993884 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:28.999430 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.006119 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:29.006717 6300 raft_consensus.cc:1275] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260620 19:59:29.000654 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:29.008239 6908 raft_consensus.cc:2393] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:28.981024 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.002048 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:29.003217 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.010401 6227 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:29.012192 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:29.014657 6301 raft_consensus.cc:1240] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []
W20260620 19:59:29.012231 6877 leader_election.cc:336] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.743s (SENT)
I20260620 19:59:29.015494 6877 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:29.012620 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:29.026885 6906 raft_consensus.cc:2410] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260620 19:59:28.917184 9417 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:28.939424 9319 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:29.029316 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.035027 6307 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:28.967628 9377 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:29.033030 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 86076.5us
I20260620 19:59:29.033270 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:29.034144 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.040755 6228 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260620 19:59:29.042320 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:29.046959 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:29.051097 6904 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260620 19:59:29.051648 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.045472 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.054883 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:29.055698 6908 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 3.
I20260620 19:59:29.057927 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.058344 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.062098 6229 raft_consensus.cc:1240] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260620 19:59:28.971982 9302 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.063639 9233 raft_consensus.cc:697] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:29.068235 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.069682 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.038369 9431 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:29.072734 6911 raft_consensus.cc:2410] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 8 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260620 19:59:29.073304 6376 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:29.075613 6373 raft_consensus.cc:2393] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 9: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:28.968525 9262 consensus_queue.cc:237] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.025898 9238 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.078109 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.085281 6379 raft_consensus.cc:2368] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7.
I20260620 19:59:29.085433 6373 raft_consensus.cc:1240] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260620 19:59:29.085604 6377 raft_consensus.cc:1240] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260620 19:59:29.070784 9417 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260620 19:59:29.047623 9319 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:29.095525 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.059865 9377 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.106701 6376 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260620 19:59:29.107705 6909 raft_consensus.cc:1275] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260620 19:59:29.116242 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 77738.5us
I20260620 19:59:29.120003 6196 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:29.077680 9366 consensus_queue.cc:1048] T aacb05a304e442a7bd73148aff5f4a66 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260620 19:59:29.099117 9347 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.128583 6379 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
W20260620 19:59:29.127374 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 88804.8us
I20260620 19:59:29.129730 6195 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:29.087854 9349 consensus_queue.cc:1059] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.039s
I20260620 19:59:29.093227 9431 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.157708 6195 leader_election.cc:400] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:59:29.158728 6195 leader_election.cc:403] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260620 19:59:29.150450 9352 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:29.159597 9442 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260620 19:59:29.160027 9427 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.179s user 0.001s sys 0.000s
I20260620 19:59:29.150776 8829 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.010s
I20260620 19:59:29.159847 9352 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:29.162809 9352 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:29.163501 9434 log.cc:927] Time spent T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
W20260620 19:59:29.171011 9436 log.cc:927] Time spent T ddb66400acd140a5b26f1a19f2abd18e P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260620 19:59:29.172726 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.174558 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:29.175925 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.176157 6301 raft_consensus.cc:2393] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 8: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:29.177106 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.177284 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.178334 6299 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:29.179404 9338 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:29.151147 9233 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.159024 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.183198 6301 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:29.151428 9376 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.190052 6908 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260620 19:59:29.256335 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.124778 9417 consensus_queue.cc:1059] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260620 19:59:29.305751 9338 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:29.380249 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:29.413131 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:29.482323 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 328798us
I20260620 19:59:29.246439 8829 consensus_queue.cc:1048] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.262s
W20260620 19:59:29.487778 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:29.473980 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 461697us
W20260620 19:59:29.475190 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 174836us
W20260620 19:59:29.483626 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 336251us
I20260620 19:59:29.456885 9238 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:29.332809 9440 consensus_queue.cc:1059] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.124s
I20260620 19:59:29.488865 6874 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:29.493083 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:29.493460 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.494583 6379 raft_consensus.cc:1240] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:29.495242 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:29.419486 9347 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:29.360298 9302 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.488937 9349 consensus_queue.cc:1243] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260620 19:59:29.419457 9443 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6.
I20260620 19:59:29.358067 9338 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:29.538445 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:29.542899 9377 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:29.577749 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:29.524165 9319 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:29.582181 6874 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:29.584064 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 85366.5us
I20260620 19:59:29.517848 9224 consensus_queue.cc:1059] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260620 19:59:29.547137 9417 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:29.589164 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
W20260620 19:59:29.591406 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:29.518793 9347 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.591516 6874 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:29.600137 6373 raft_consensus.cc:1240] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:29.583920 9440 raft_consensus.cc:3055] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:29.604549 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.609241 6299 raft_consensus.cc:1275] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260620 19:59:29.611730 6373 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260620 19:59:29.623906 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:29.616120 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:29.596395 9467 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 19:59:29.624789 6906 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:29.602293 9240 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:29.635715 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:29.589113 9319 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
W20260620 19:59:29.640514 6195 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:29.640928 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.645035 6196 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:29.647279 9409 log.cc:927] Time spent T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 1.502s user 0.009s sys 0.003s
I20260620 19:59:29.633421 9302 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:29.648777 9450 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:29.604478 9440 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:59:29.650486 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 59029.7us
I20260620 19:59:29.588785 9352 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.656774 6301 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.658537 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:29.662521 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:29.663380 6373 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:29.601468 9348 consensus_queue.cc:1059] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260620 19:59:29.636428 9431 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:29.596884 9417 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.663630 9377 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
W20260620 19:59:29.665781 9447 log.cc:927] Time spent T 65e57f1c5f9f437db061394645beec68 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20260620 19:59:29.666381 9446 log.cc:927] Time spent T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260620 19:59:29.636907 9467 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:29.666325 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 2011) took 1776 ms. Trace:
I20260620 19:59:29.630543 9224 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.033s
I20260620 19:59:29.667050 9041 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:29.663937 9450 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.713965 9450 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.668143 9135 raft_consensus.cc:3055] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:29.668423 9352 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.689201 9377 consensus_queue.cc:1243] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260620 19:59:29.691545 9386 raft_consensus.cc:3055] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:29.694939 9347 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.695390 6875 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:29.695128 9440 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.674345 9468 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:29.706975 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:29.709321 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.707842 6230 rpcz_store.cc:276] 0620 19:59:27.890010 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:27.890531 (+ 521us) service_pool.cc:224] Handling call
0620 19:59:27.936996 (+ 46465us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:27.964256 (+ 27260us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:27.964302 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:27.973756 (+ 9454us) log.cc:844] Serialized 26 byte log entry
0620 19:59:28.136416 (+162660us) raft_consensus.cc:1645] Marking committed up to 2
0620 19:59:28.136453 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:28.136512 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:29.646943 (+1510431us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b24000f5528. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b564f614 00007fa6b564d71b 00007fa6b5808619 00007fa6b58082a5 00007fa6b58078ed 00007fa6b843d4ae 00007fa6b8454c2c 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1
0620 19:59:29.666007 (+ 19064us) raft_consensus.cc:1685] finished
0620 19:59:29.666054 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:29.666251 (+ 197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22117,"raft.queue_time_us":25723,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":6201,"spinlock_wait_cycles":10896256,"thread_start_us":138305,"threads_started":1,"wal-append.queue_time_us":160841}
I20260620 19:59:29.747080 6226 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:29.710813 9467 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260620 19:59:29.711031 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.711236 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.711808 9464 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:29.677999 9262 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:29.655143 9240 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.720009 9342 raft_consensus.cc:3055] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Stepping down as leader of term 9
I20260620 19:59:29.734838 9386 raft_consensus.cc:740] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:29.768139 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.716871 9135 raft_consensus.cc:740] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:29.723373 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1754) took 1397 ms. Trace:
I20260620 19:59:29.769062 6374 rpcz_store.cc:276] 0620 19:59:28.325845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:28.337282 (+ 11437us) service_pool.cc:224] Handling call
0620 19:59:28.337573 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:28.528291 (+190718us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:28.528328 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:28.528366 (+ 38us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:28.528401 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:29.723139 (+1194738us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400063954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:29.723192 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:29.723310 (+ 118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438067072}
I20260620 19:59:29.769927 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.658839 9463 consensus_queue.cc:1059] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260620 19:59:29.662249 9302 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:29.675974 9470 consensus_queue.cc:1059] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260620 19:59:29.728901 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:29.729007 9430 raft_consensus.cc:3055] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:29.730193 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.732380 9445 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:29.759315 9262 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:29.749778 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 97695.1us
I20260620 19:59:29.755620 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.790985 6380 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 8: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:29.726974 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:29.778018 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 62743us
I20260620 19:59:29.791695 6307 raft_consensus.cc:2410] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260620 19:59:29.792057 6877 leader_election.cc:336] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.390s (SENT)
I20260620 19:59:29.792443 6195 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:29.793051 6877 leader_election.cc:336] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.917s (SENT)
I20260620 19:59:29.793557 6877 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:29.734181 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 757) took 1477 ms. Trace:
I20260620 19:59:29.788179 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.759088 9128 consensus_queue.cc:1059] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
W20260620 19:59:29.785710 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 140770us
I20260620 19:59:29.781864 9440 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:29.795135 6303 rpcz_store.cc:276] 0620 19:59:28.256238 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:28.348810 (+ 92572us) service_pool.cc:224] Handling call
0620 19:59:28.410245 (+ 61435us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:28.695086 (+284841us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:29.733937 (+1038851us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54003b4a54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:29.734097 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623321856}
I20260620 19:59:29.795646 6373 raft_consensus.cc:2393] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:29.767977 9342 raft_consensus.cc:740] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
W20260620 19:59:29.801568 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 106220us
I20260620 19:59:29.805711 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.807256 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:29.808064 6306 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:29.810417 6230 raft_consensus.cc:2410] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:29.784700 9430 raft_consensus.cc:740] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:29.815184 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.816362 9463 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:29.817384 9468 raft_consensus.cc:3055] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:29.817138 9463 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.817919 9468 raft_consensus.cc:740] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:29.819159 9468 consensus_queue.cc:260] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.820727 9468 raft_consensus.cc:3060] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
W20260620 19:59:29.823235 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:29.824952 6877 leader_election.cc:336] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.515s (SENT)
I20260620 19:59:29.825358 6877 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:29.826699 9475 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:29.826881 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.828267 6300 raft_consensus.cc:1240] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260620 19:59:29.828303 9475 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:29.829610 6300 raft_consensus.cc:1240] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260620 19:59:29.831529 6875 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:29.832702 9342 consensus_queue.cc:260] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:29.833706 9468 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:29.833899 9342 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:29.823052 9470 raft_consensus.cc:3055] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:29.834682 9041 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:29.834870 9470 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:29.835207 9041 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.835767 9470 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.836963 9470 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:29.837297 9041 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.839046 9302 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:29.834272 9468 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:29.839557 9302 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:29.839877 9468 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:29.840076 9302 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.842741 9464 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:29.843163 9464 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:29.843488 9464 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:29.851241 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.851940 6303 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:29.794554 9262 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:29.797065 9460 raft_consensus.cc:2749] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:29.858659 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.860622 6303 raft_consensus.cc:2393] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:29.864504 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.865157 6380 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:29.854108 9302 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.871871 9477 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:29.872543 9240 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:29.872427 9477 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.873224 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.873824 6379 raft_consensus.cc:2468] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:29.874578 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.874936 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.875211 6379 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 9.
I20260620 19:59:29.875854 6374 raft_consensus.cc:2410] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260620 19:59:29.876135 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:29.876701 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.877485 6379 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 6: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:29.877453 6374 raft_consensus.cc:2468] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:29.875077 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:29.878700 6373 raft_consensus.cc:2410] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260620 19:59:29.879616 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.794358 9445 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.409912 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 840) took 1385 ms. Trace:
I20260620 19:59:29.944063 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.938810 9463 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:59:30.131523 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 831) took 1288 ms. Trace:
I20260620 19:59:29.891346 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:30.051115 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.423756 6307 pending_rounds.cc:85] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 3
I20260620 19:59:30.433805 6307 pending_rounds.cc:107] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260620 19:59:30.434062 6307 raft_consensus.cc:2889] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:29.891119 9417 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.434624 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:29.918859 9481 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.080811 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:30.447237 6226 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:30.414500 6304 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 745) took 2200 ms (client timeout 1589 ms). Trace:
I20260620 19:59:29.941134 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:29.950160 9262 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.415623 6195 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:30.449235 6304 rpcz_store.cc:269] 0620 19:59:28.213964 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:28.220228 (+ 6264us) service_pool.cc:224] Handling call
0620 19:59:30.151763 (+1931535us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54003a6954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:30.414386 (+262623us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":457325952}
W20260620 19:59:30.056105 9480 log.cc:927] Time spent T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260620 19:59:30.418216 6344 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:30.124643 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:30.118871 9344 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.107599 9365 raft_consensus.cc:2955] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 3 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:30.113083 9338 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:30.070902 9386 consensus_queue.cc:260] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.137955 6198 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:29.942804 6306 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:29.917891 9468 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.913084 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:30.151127 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53406 (request call id 888) took 1166 ms. Trace:
I20260620 19:59:30.100517 9408 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:29.998994 9464 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.068455 9135 consensus_queue.cc:260] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.424551 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:30.536602 6379 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 9.
I20260620 19:59:29.937443 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:30.432085 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 762) took 1259 ms. Trace:
I20260620 19:59:30.411593 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 2028) took 1566 ms. Trace:
I20260620 19:59:29.942103 6223 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 2024) took 1236 ms. Trace:
W20260620 19:59:30.342769 6347 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.003s sys 0.016s
I20260620 19:59:30.434952 9417 consensus_queue.cc:1059] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:59:29.923075 6230 raft_consensus.cc:1240] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: []
W20260620 19:59:30.561733 6347 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.767s user 0.003s sys 0.019s
W20260620 19:59:30.074836 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.143141 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:30.428308 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.409510 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 830) took 1566 ms. Trace:
I20260620 19:59:30.410446 6904 rpcz_store.cc:276] 0620 19:59:29.024955 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.059703 (+ 34748us) service_pool.cc:224] Handling call
0620 19:59:29.074423 (+ 14720us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:29.114355 (+ 39932us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:29.114387 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:29.114427 (+ 40us) raft_consensus.cc:1645] Marking committed up to 11
0620 19:59:29.114460 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:30.082932 (+968472us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b54006f8e54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:30.409741 (+326809us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:30.409860 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91370880}
I20260620 19:59:30.445667 6909 rpcz_store.cc:276] 0620 19:59:28.843482 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:28.844241 (+ 759us) service_pool.cc:224] Handling call
0620 19:59:28.863893 (+ 19652us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:29.109251 (+245358us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:30.131220 (+1021969us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54006f9ad4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:30.131446 (+ 226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":559892864}
I20260620 19:59:30.413909 9406 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:30.563055 6902 rpcz_store.cc:276] 0620 19:59:28.843142 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:28.843597 (+ 455us) service_pool.cc:224] Handling call
0620 19:59:28.843911 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:29.066517 (+222606us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:29.066549 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:29.066587 (+ 38us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:29.066648 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:30.093302 (+1026654us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400631ad4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:30.093384 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:30.409429 (+316045us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511401344}
W20260620 19:59:30.079355 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 252999us
I20260620 19:59:30.448740 6231 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
W20260620 19:59:30.568449 6231 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 2027) took 1724 ms (client timeout 2199 ms). Trace:
I20260620 19:59:30.443189 6874 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:30.496539 6299 raft_consensus.cc:1240] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:30.496951 9481 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:29.947906 9388 raft_consensus.cc:3055] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:30.499295 9262 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:30.503726 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 85602.1us
I20260620 19:59:30.504616 6908 rpcz_store.cc:276] 0620 19:59:28.984501 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.056491 (+ 71990us) service_pool.cc:224] Handling call
0620 19:59:29.056923 (+ 432us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:29.194383 (+137460us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:30.150821 (+956438us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540054fbd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:30.151067 (+ 246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305578752}
W20260620 19:59:30.517624 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 379710us
W20260620 19:59:30.144992 9426 log.cc:927] Time spent T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 1.375s user 0.004s sys 0.000s
W20260620 19:59:29.948583 9005 consensus_meta.cc:287] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 1.148s user 0.012s sys 0.001s
I20260620 19:59:30.563542 6379 raft_consensus.cc:2955] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 3 to 6, 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } } }
I20260620 19:59:29.874507 9477 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.546788 6305 rpcz_store.cc:276] 0620 19:59:29.172827 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.178683 (+ 5856us) service_pool.cc:224] Handling call
0620 19:59:29.943923 (+765240us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b5800003098. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b5791371 00007fa6b5791222 00007fa6be6ae7d9 00007fa6bd8095e0 00007fa6bd8d27ab 00007fa6bd8d2831 00007fa6bd8807a0 00007fa6bd880084 00007fa6b6df8834 00007fa6b6df87c2
0620 19:59:30.432032 (+488109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8941312}
I20260620 19:59:30.547596 6225 rpcz_store.cc:276] 0620 19:59:28.846154 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:30.136457 (+1290303us) service_pool.cc:224] Handling call
0620 19:59:30.410965 (+274508us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:30.411156 (+ 191us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:30.411193 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:30.411252 (+ 59us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:30.411301 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:30.411375 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:30.411500 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:30.576668 8864 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:30.577514 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:30.566772 9369 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.578207 9354 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.578701 9135 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:30.578899 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:30.585187 9381 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:30.568917 6231 rpcz_store.cc:269] 0620 19:59:28.844530 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.940863 (+1096333us) service_pool.cc:224] Handling call
0620 19:59:30.568295 (+627432us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:30.209095 9377 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:29.985834 9460 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:29.913973 9453 raft_consensus.cc:3055] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
W20260620 19:59:30.580744 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 137634us
I20260620 19:59:30.584831 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:30.251727 8829 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:30.594107 6300 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.004s sys 0.032s
W20260620 19:59:30.632879 6300 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.761s user 0.004s sys 0.032s
I20260620 19:59:30.548444 6223 rpcz_store.cc:276] 0620 19:59:28.709054 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:28.826292 (+117238us) service_pool.cc:224] Handling call
0620 19:59:28.839717 (+ 13425us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:28.863334 (+ 23617us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:28.863366 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:28.967407 (+104041us) log.cc:844] Serialized 26 byte log entry
0620 19:59:29.344702 (+377295us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:29.344758 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:29.941689 (+596931us) spinlock_profiling.cc:243] Waited 9.36 ms on lock 0x7b5400286754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:29.941738 (+ 49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:29.941791 (+ 53us) raft_consensus.cc:1685] finished
0620 19:59:29.941808 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:29.942041 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19903,"raft.queue_time_us":14722,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":97071,"spinlock_wait_cycles":21529472,"thread_start_us":278768,"threads_started":1,"wal-append.queue_time_us":462156,"wal-append.run_cpu_time_us":1571,"wal-append.run_wall_time_us":1001591}
I20260620 19:59:30.096249 9351 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:30.601145 6911 raft_consensus.cc:3060] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:30.649653 9351 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.651906 9351 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.652408 9351 consensus_queue.cc:1048] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 19:59:29.918865 9467 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.653141 9338 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.653673 9338 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.655066 9467 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.655709 9338 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:30.612442 6906 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:30.656581 9377 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.658941 9377 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.659979 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:30.661237 6380 raft_consensus.cc:1240] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:30.613160 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:30.663059 9475 raft_consensus.cc:3055] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:30.663465 9475 raft_consensus.cc:740] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:30.663650 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2053) took 2507 ms. Trace:
I20260620 19:59:30.664288 9475 consensus_queue.cc:260] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.664002 6378 rpcz_store.cc:276] 0620 19:59:28.155821 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:28.171799 (+ 15978us) service_pool.cc:224] Handling call
0620 19:59:28.172118 (+ 319us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:28.652673 (+480555us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:28.652717 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:28.687501 (+ 34784us) log.cc:844] Serialized 26 byte log entry
0620 19:59:28.732238 (+ 44737us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:28.732287 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:28.733576 (+ 1289us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:30.144194 (+1410618us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:30.663362 (+519168us) raft_consensus.cc:1685] finished
0620 19:59:30.663383 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:30.663576 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3520,"raft.queue_time_us":34144,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":3925760,"thread_start_us":524460,"threads_started":2,"wal-append.queue_time_us":81946}
I20260620 19:59:30.665321 6904 raft_consensus.cc:1275] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260620 19:59:30.580385 9358 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Leader election won for term 4
I20260620 19:59:30.666221 9358 raft_consensus.cc:697] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:30.667327 9358 consensus_queue.cc:237] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.624037 6909 raft_consensus.cc:2393] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:30.579733 6377 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:30.614025 6910 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:30.595656 9386 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:30.585850 9388 raft_consensus.cc:740] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:29.951166 9430 consensus_queue.cc:260] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:30.640497 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:30.724098 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.600236 9487 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.566998 9408 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.596554 9440 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.617251 9477 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.631901 9347 consensus_queue.cc:1048] T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260620 19:59:30.645704 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:30.741417 6374 raft_consensus.cc:2468] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:30.632056 9453 raft_consensus.cc:740] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:30.648319 8829 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.665424 9475 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
W20260620 19:59:30.758819 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.620638 9262 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260620 19:59:30.679129 6909 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260620 19:59:30.679840 6904 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:30.760056 6903 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:30.760640 9339 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.761085 9339 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:30.762326 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.598263 9429 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:30.567814 9344 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.764484 9344 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.685501 6911 raft_consensus.cc:2468] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:30.726248 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:30.765363 6376 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:30.728901 6910 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260620 19:59:30.734586 9487 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.766964 8864 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:30.767397 8864 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:30.767606 9487 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.768236 9487 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.768672 9487 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:30.582906 9352 consensus_queue.cc:1175] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "39662d01be7a44f682d160227f457661" responder_term: 9 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 8. Current term is 9. Ops: [8.4-8.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20260620 19:59:30.722998 9390 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.771696 9390 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.735543 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 846) took 1143 ms. Trace:
I20260620 19:59:30.630290 9381 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.772809 6906 rpcz_store.cc:276] 0620 19:59:29.585053 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.654210 (+ 69157us) service_pool.cc:224] Handling call
0620 19:59:30.724428 (+1070218us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400176bd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:30.728732 (+ 4304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2188439296}
I20260620 19:59:30.773968 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:30.774483 6906 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:30.774519 9381 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.775426 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:30.729492 9388 consensus_queue.cc:260] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.776429 9388 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:30.775455 6874 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:30.630780 9417 raft_consensus.cc:3055] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:30.705744 9367 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.623292 9369 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.745397 9477 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.771191 9319 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.626147 9354 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.743161 6374 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:30.731055 9240 consensus_queue.cc:1059] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260620 19:59:30.796515 6906 raft_consensus.cc:1275] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260620 19:59:30.797461 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 229450us
I20260620 19:59:30.679771 9460 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:30.781097 9383 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.782392 9417 raft_consensus.cc:740] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:30.783378 9429 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:30.816563 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.750732 9440 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.819211 9369 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:30.817196 6877 leader_election.cc:336] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.589s (SENT)
I20260620 19:59:30.827338 9481 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.830351 9477 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.831241 9464 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:30.790562 9367 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.839185 9429 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.839931 9383 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.843020 9453 consensus_queue.cc:260] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.844077 9468 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:30.845495 9354 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.810002 9240 consensus_queue.cc:1243] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Peer 39662d01be7a44f682d160227f457661 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260620 19:59:30.803694 9470 consensus_meta.cc:287] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.966s user 0.010s sys 0.000s
I20260620 19:59:30.799649 9319 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.852649 6877 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:30.858809 9408 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.859156 8829 leader_election.cc:290] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.865069 9262 consensus_queue.cc:1048] T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260620 19:59:30.865684 6196 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:30.868696 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 847) took 1252 ms. Trace:
I20260620 19:59:30.869849 6910 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260620 19:59:30.871618 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 849) took 1041 ms. Trace:
I20260620 19:59:30.901136 6910 pending_rounds.cc:85] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting all ops after (but not including) 7
I20260620 19:59:30.901427 6910 pending_rounds.cc:107] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260620 19:59:30.901288 6911 rpcz_store.cc:276] 0620 19:59:29.830169 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.912810 (+ 82641us) service_pool.cc:224] Handling call
0620 19:59:30.871453 (+958643us) spinlock_profiling.cc:243] Waited 96.5 ms on lock 0x7b54005014d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:30.871545 (+ 92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":221901568}
I20260620 19:59:30.901803 6910 raft_consensus.cc:2889] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260620 19:59:30.875468 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.901281 9468 raft_consensus.cc:2749] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:30.900705 6905 rpcz_store.cc:276] 0620 19:59:29.616783 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.712049 (+ 95266us) service_pool.cc:224] Handling call
0620 19:59:30.151024 (+438975us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:30.151398 (+ 374us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:30.151426 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:30.151892 (+ 466us) log.cc:844] Serialized 26 byte log entry
0620 19:59:30.593921 (+442029us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:30.593957 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:30.594025 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:30.824225 (+230200us) raft_consensus.cc:1685] finished
0620 19:59:30.824254 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:30.868407 (+ 44153us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b540054fbd0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:30.868627 (+ 220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1080,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":86,"spinlock_wait_cycles":1008657664,"thread_start_us":441656,"threads_started":1,"wal-append.queue_time_us":587079}
I20260620 19:59:30.877748 9466 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:30.804517 9487 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:30.893070 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 75887.7us
W20260620 19:59:30.914059 6877 leader_election.cc:336] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.271s (SENT)
I20260620 19:59:30.914536 6877 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:30.894573 6271 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:30.900023 6904 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260620 19:59:30.879503 9477 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.908021 6196 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:30.879904 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 291450us
I20260620 19:59:30.924207 6196 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:30.897450 9481 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:30.927204 6271 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:30.916325 9477 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.928102 9453 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:30.929916 9477 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.922923 9470 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:30.930482 9367 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.931561 9470 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:30.931910 9383 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.932058 9470 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.932554 9417 consensus_queue.cc:260] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.935081 9429 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.926329 9408 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:30.934350 9470 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.938524 9408 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260620 19:59:30.939950 9319 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:30.942216 9460 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:30.939971 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:30.950950 9233 raft_consensus.cc:2749] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:30.952615 9493 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:30.953176 9493 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:30.953395 9240 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:30.953576 9493 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:30.953878 9240 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:30.954316 9240 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:30.913664 8829 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:30.958233 9481 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.958726 9390 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:30.961429 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 545763us
I20260620 19:59:30.949460 6196 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:30.964527 6271 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:30.960882 9491 log.cc:927] Time spent T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.209s user 0.000s sys 0.002s
I20260620 19:59:30.970993 6196 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:30.973349 6271 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:30.975312 6271 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:30.985502 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 196146us
I20260620 19:59:30.987231 6375 pending_rounds.cc:85] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 6
I20260620 19:59:30.987581 6375 pending_rounds.cc:107] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260620 19:59:30.987843 6375 raft_consensus.cc:2889] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:30.968116 9390 raft_consensus.cc:2749] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:30.948486 8864 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:30.972050 9483 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:30.987112 9351 raft_consensus.cc:2804] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:30.987461 9408 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:30.988095 9354 raft_consensus.cc:2749] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260620 19:59:31.025529 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:31.015316 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 95239.8us
I20260620 19:59:31.027096 6875 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:30.984504 9236 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260620 19:59:30.969218 9233 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260620 19:59:31.023725 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35548 (request call id 503) took 1189 ms. Trace:
I20260620 19:59:31.029800 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:31.031139 6198 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.034286 6230 rpcz_store.cc:276] 0620 19:59:29.834065 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.858513 (+ 24448us) service_pool.cc:224] Handling call
0620 19:59:29.858822 (+ 309us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:30.562052 (+703230us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:31.023455 (+461403us) spinlock_profiling.cc:243] Waited 64.1 ms on lock 0x7b5400245c54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:31.023653 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":147352448}
W20260620 19:59:31.036092 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:31.043319 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:30.988476 9133 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.037s
W20260620 19:59:31.048440 6198 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.018021 9483 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:31.051543 6230 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260620 19:59:31.053977 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.061208 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:31.063515 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.025164 9408 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:31.068234 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.069661 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.065420 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:31.072185 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:31.073567 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.030194 9429 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:31.028307 9351 raft_consensus.cc:697] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:31.044479 9445 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.083487 6223 raft_consensus.cc:1240] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:31.049984 9477 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:31.018646 9379 consensus_queue.cc:1048] T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260620 19:59:31.048620 9487 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.055734 9358 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.058388 9483 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:31.062556 9440 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.077028 9408 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:31.062130 9481 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.049296 9367 consensus_queue.cc:1059] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260620 19:59:31.099182 6377 raft_consensus.cc:1275] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260620 19:59:31.116892 6223 raft_consensus.cc:1240] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
W20260620 19:59:31.099088 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 83697.5us
I20260620 19:59:31.106163 9406 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
I20260620 19:59:31.102353 9349 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.106202 9470 consensus_queue.cc:1059] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260620 19:59:31.131016 6380 raft_consensus.cc:1275] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260620 19:59:31.133116 6877 leader_election.cc:336] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 2.030s (SENT)
I20260620 19:59:31.135267 6877 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:31.085876 9509 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.049s
I20260620 19:59:31.143319 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.144688 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.133821 9349 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:31.098738 9477 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.102821 9487 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.156498 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.157358 6302 raft_consensus.cc:2393] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 8: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:31.158316 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.159963 6307 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260620 19:59:31.160393 6307 pending_rounds.cc:85] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 4
I20260620 19:59:31.160630 6307 pending_rounds.cc:107] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260620 19:59:31.160879 6307 raft_consensus.cc:2889] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:31.108683 9358 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:31.164901 6903 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 854) took 1324 ms (client timeout 1572 ms). Trace:
W20260620 19:59:31.167127 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 116203us
I20260620 19:59:31.132411 9406 consensus_queue.cc:1243] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260620 19:59:31.176138 6903 rpcz_store.cc:269] 0620 19:59:29.840521 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.937107 (+ 96586us) service_pool.cc:224] Handling call
0620 19:59:31.164743 (+1227636us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400700dd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:31.164849 (+ 106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511018240}
I20260620 19:59:31.146154 9294 raft_consensus.cc:3055] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:31.154611 9440 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:31.145591 9236 raft_consensus.cc:3055] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:31.203016 9236 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:31.143841 9351 consensus_queue.cc:237] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.203786 9236 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.154801 9369 consensus_queue.cc:1048] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20260620 19:59:31.186307 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:31.206569 6298 raft_consensus.cc:2393] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 4: Already voted for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in this term.
I20260620 19:59:31.162544 9390 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260620 19:59:31.193123 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.187532 9477 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.188416 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.198143 9342 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:31.198401 9503 log.cc:927] Time spent T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
W20260620 19:59:31.198807 9504 log.cc:927] Time spent T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260620 19:59:31.199159 6906 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260620 19:59:31.220245 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.199646 6304 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260620 19:59:31.200024 6905 raft_consensus.cc:1275] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260620 19:59:31.200335 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.221825 6306 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 4.
I20260620 19:59:31.200776 6223 raft_consensus.cc:1275] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:31.200868 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:31.187295 9497 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:59:31.226583 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.226907 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.227674 6910 raft_consensus.cc:2410] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260620 19:59:31.191825 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.189677 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:31.192525 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.227667 6232 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:31.231539 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:31.232276 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.213810 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:31.232614 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:31.215900 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:31.198449 9501 log.cc:927] Time spent T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20260620 19:59:31.234150 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.232920 6303 raft_consensus.cc:2368] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7.
I20260620 19:59:31.207626 9294 raft_consensus.cc:740] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:31.147964 9262 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:31.235728 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.247876 6379 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:31.236145 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.236941 6906 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:31.237362 9493 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:31.239625 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:31.240947 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.235180 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.251494 6908 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 4.
I20260620 19:59:31.213574 9440 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.247066 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.249008 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.257400 6379 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:31.260036 6379 raft_consensus.cc:1275] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:31.252785 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.238462 8829 raft_consensus.cc:2749] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:31.182248 9518 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.094s
I20260620 19:59:31.268605 6230 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:31.269389 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.220399 9342 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:31.305258 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.309082 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.331390 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.332286 6377 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:31.252333 9262 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:31.314870 9466 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.337436 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:31.330631 9426 log.cc:927] Time spent T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260620 19:59:31.264019 9440 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.348083 6303 raft_consensus.cc:1240] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
W20260620 19:59:31.350630 9515 log.cc:927] Time spent T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260620 19:59:31.352098 6300 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1715) took 1556 ms. Trace:
W20260620 19:59:31.371263 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 61072.8us
W20260620 19:59:31.353492 9475 consensus_meta.cc:287] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.595s user 0.010s sys 0.000s
I20260620 19:59:31.353757 9466 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.354661 9262 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.356825 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:31.360042 6877 leader_election.cc:336] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701): Timed out: RequestConsensusVote RPC to 127.4.244.131:41701 timed out after 1.866s (SENT)
I20260620 19:59:31.322266 8829 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:31.369961 6231 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:31.371546 6300 rpcz_store.cc:276] 0620 19:59:29.795616 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:29.830274 (+ 34658us) service_pool.cc:224] Handling call
0620 19:59:29.830539 (+ 265us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:29.861768 (+ 31229us) raft_consensus.cc:1534] Early marking committed up to index 3
0620 19:59:30.633260 (+771492us) log.cc:844] Serialized 14 byte log entry
0620 19:59:30.882044 (+248784us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:30.895769 (+ 13725us) log.cc:844] Serialized 26 byte log entry
0620 19:59:30.908620 (+ 12851us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:30.908676 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:30.908841 (+ 165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:31.351921 (+443080us) raft_consensus.cc:1685] finished
0620 19:59:31.351938 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:31.352047 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":294,"prepare.queue_time_us":433258,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":422,"raft.queue_time_us":8499,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":14868,"thread_start_us":994002,"threads_started":2,"wal-append.queue_time_us":369755}
W20260620 19:59:31.382850 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.388116 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.415035 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.410432 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:31.387358 9517 log.cc:927] Time spent T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260620 19:59:31.414752 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.425647 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.428231 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92212144534e4394844003016ba67a47" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:31.429764 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 69757.9us
W20260620 19:59:31.430063 6381 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 615) took 1555 ms (client timeout 2055 ms). Trace:
W20260620 19:59:31.433221 6381 rpcz_store.cc:269] 0620 19:59:29.880013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:30.659746 (+779733us) service_pool.cc:224] Handling call
0620 19:59:31.429901 (+770155us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54000eae54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:31.430009 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":800004992}
I20260620 19:59:31.415616 6377 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:31.330358 9294 consensus_queue.cc:260] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.443799 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.408176 9442 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.440604 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:31.442016 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.452191 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.452852 6302 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:31.437533 9475 raft_consensus.cc:2749] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260620 19:59:31.444289 9386 consensus_meta.cc:287] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.767s user 0.008s sys 0.004s
I20260620 19:59:31.436079 9349 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.460198 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.437167 9487 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.399860 9240 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.424954 8829 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:31.473541 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 247070us
I20260620 19:59:31.475266 6299 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
W20260620 19:59:31.481710 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.483763 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:31.484439 6377 raft_consensus.cc:3055] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:31.484882 6377 raft_consensus.cc:740] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:31.487502 9477 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20260620 19:59:31.450203 9353 consensus_queue.cc:1048] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260620 19:59:31.493443 9440 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.454053 9442 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:31.495074 9388 consensus_meta.cc:287] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.718s user 0.010s sys 0.001s
I20260620 19:59:31.496543 9442 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.457506 9358 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:31.497814 9453 consensus_meta.cc:287] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.569s user 0.010s sys 0.000s
I20260620 19:59:31.467939 9339 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:31.472770 9262 consensus_queue.cc:1048] T 1cb7a18e80e145f6913b635487750934 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260620 19:59:31.499353 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:31.499944 6299 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:31.503592 9466 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.485742 6377 consensus_queue.cc:260] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.506062 6377 raft_consensus.cc:3060] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:31.507459 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.508351 6904 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:31.510179 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.515393 6227 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260620 19:59:31.515851 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:31.518381 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.518652 9408 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.520944 9408 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.522781 6301 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:31.524017 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.522293 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:31.526988 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.527642 6302 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 10.
W20260620 19:59:31.529237 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:31.498416 9529 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260620 19:59:31.541718 6299 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:31.535730 6377 raft_consensus.cc:1275] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260620 19:59:31.537585 6196 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:31.560930 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:31.562395 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:31.549198 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.563251 6299 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:31.563578 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:31.564111 6910 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:31.564781 6377 pending_rounds.cc:85] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting all ops after (but not including) 6
I20260620 19:59:31.551260 6874 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:31.567953 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.566092 6377 pending_rounds.cc:107] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260620 19:59:31.568792 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:31.569438 6903 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:31.569118 6377 raft_consensus.cc:2889] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:31.565577 6195 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:31.571545 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.573369 6372 raft_consensus.cc:1275] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260620 19:59:31.578521 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.584707 6376 raft_consensus.cc:1275] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260620 19:59:31.572579 6273 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:31.587896 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.588725 6377 raft_consensus.cc:2393] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 8: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:31.573503 6874 leader_election.cc:400] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7.
I20260620 19:59:31.576049 6196 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:31.589608 6874 leader_election.cc:403] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260620 19:59:31.574112 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.591499 6299 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:31.591677 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.547935 9386 raft_consensus.cc:2764] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260620 19:59:31.596928 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.595479 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.541381 9429 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:31.595693 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:31.562247 9525 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:31.497852 9483 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.574558 8829 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.583370 9349 raft_consensus.cc:2749] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260620 19:59:31.583361 9442 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:31.610359 6906 raft_consensus.cc:1240] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
W20260620 19:59:31.595801 9528 log.cc:927] Time spent T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260620 19:59:31.596236 9499 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:31.594113 9342 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.570766 9538 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.575778 9466 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:31.567524 9379 consensus_queue.cc:1048] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20260620 19:59:31.596549 9453 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 6. Current term is 7.
W20260620 19:59:31.610033 9520 log.cc:927] Time spent T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
I20260620 19:59:31.608418 9365 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:31.620796 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.567253 9383 consensus_queue.cc:1059] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260620 19:59:31.599130 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 153224us
I20260620 19:59:31.598968 9530 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.622457 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:31.661861 6903 raft_consensus.cc:2468] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:31.662563 6904 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:31.621800 9499 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:31.662955 6909 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
W20260620 19:59:31.646394 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.619647 6224 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:31.614085 9442 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
W20260620 19:59:31.631628 6876 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.677523 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.623411 9483 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.623852 9466 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:31.608966 9525 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:31.658087 9365 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:31.698729 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.623396 9523 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260620 19:59:31.662166 9339 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.674294 9499 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:31.676817 9442 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:31.701188 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 123085us
I20260620 19:59:31.705722 6195 leader_election.cc:304] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:31.707212 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.707995 6377 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 9.
I20260620 19:59:31.690912 9466 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:31.691272 9430 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:31.700793 9365 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:31.735682 9460 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:31.743535 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.747654 6904 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:31.750105 6375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 613) took 1876 ms. Trace:
I20260620 19:59:31.750556 6375 rpcz_store.cc:276] 0620 19:59:29.874105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:30.968976 (+1094871us) service_pool.cc:224] Handling call
0620 19:59:30.987052 (+ 18076us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:30.988304 (+ 1252us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:30.988331 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:31.003521 (+ 15190us) log.cc:844] Serialized 26 byte log entry
0620 19:59:31.437846 (+434325us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:31.452883 (+ 15037us) log.cc:844] Serialized 14 byte log entry
0620 19:59:31.749810 (+296927us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:31.749879 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:31.749920 (+ 41us) raft_consensus.cc:1685] finished
0620 19:59:31.749934 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:31.750053 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76196,"prepare.queue_time_us":15629,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":162291,"thread_start_us":630761,"threads_started":2,"wal-append.queue_time_us":509106}
W20260620 19:59:31.751765 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 56206.9us
W20260620 19:59:31.757290 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:31.758275 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 52635.1us
I20260620 19:59:31.628919 9538 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:31.709761 9339 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:31.736285 9460 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:31.764926 9541 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260620 19:59:31.767957 9236 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:31.771559 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:31.826975 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 231329us
I20260620 19:59:31.771699 9525 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:31.779628 8864 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.681154 9530 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:31.784008 6198 leader_election.cc:304] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:31.786049 6195 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:31.786365 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 178230us
I20260620 19:59:31.788410 9538 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:31.792508 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.762683 9445 raft_consensus.cc:2749] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:31.801397 9294 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:31.802475 6376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.803833 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.804005 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:31.854898 6902 raft_consensus.cc:2393] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 4: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:31.801431 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.804837 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.855880 6373 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:31.856045 6877 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:31.806974 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:31.818616 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 225129us
I20260620 19:59:31.839561 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.850400 6271 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:31.806161 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.857191 6877 leader_election.cc:304] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:31.858379 6877 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:31.862210 6877 leader_election.cc:304] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:31.862457 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 78530us
I20260620 19:59:31.863291 6198 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:31.864528 6198 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:31.858733 9429 raft_consensus.cc:2749] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:31.847863 9383 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260620 19:59:31.876977 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 90989.4us
I20260620 19:59:31.880633 6877 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:31.891402 6195 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:31.881753 6910 raft_consensus.cc:2468] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:31.876085 6196 leader_election.cc:304] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:31.823863 9523 consensus_queue.cc:1048] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260620 19:59:31.895148 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.952620 6307 raft_consensus.cc:2410] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:31.897224 9354 consensus_queue.cc:1048] T 6d08b9c2da1448139f24e1d94c9616f3 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260620 19:59:31.898072 9369 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260620 19:59:31.899472 9518 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:31.900251 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:31.901461 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:31.962749 6300 raft_consensus.cc:2393] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:31.902437 6874 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
W20260620 19:59:31.963294 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 60914us
I20260620 19:59:31.904311 6195 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:31.873941 8864 raft_consensus.cc:2749] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260620 19:59:31.914871 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.915722 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:31.889120 9445 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:31.967731 6910 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:31.917840 9550 raft_consensus.cc:2804] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:31.923782 6196 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:31.953974 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 62133.1us
I20260620 19:59:31.991752 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.992024 6877 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:31.955972 6907 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260620 19:59:31.861478 9453 raft_consensus.cc:2804] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260620 19:59:31.957578 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.870864 9440 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:31.913169 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:31.893136 9352 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:31.888675 9319 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260620 19:59:31.955000 9541 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260620 19:59:31.888437 9383 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:31.893970 9429 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260620 19:59:31.970326 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:31.973493 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:32.011386 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 107123us
I20260620 19:59:31.975392 9493 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.980438 9240 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:31.987883 9379 raft_consensus.cc:3055] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:31.951601 9133 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260620 19:59:31.995463 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 145128us
I20260620 19:59:31.888465 9497 consensus_queue.cc:1048] T 72968f173fdb4542920e97a940db9668 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.081s
I20260620 19:59:31.953186 9464 raft_consensus.cc:2749] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 19:59:31.963940 9408 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:32.007526 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:32.011904 6877 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:32.066124 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 54280.8us
I20260620 19:59:32.067101 6877 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:31.968541 9475 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:32.012802 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.013660 6875 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:32.049823 6344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:31.998970 9453 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:32.055763 6271 leader_election.cc:304] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.004127 9468 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:32.004397 9352 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:32.053411 6227 pending_rounds.cc:85] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Aborting all ops after (but not including) 5
W20260620 19:59:32.070820 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 58804.5us
I20260620 19:59:32.075178 6227 pending_rounds.cc:107] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260620 19:59:32.015358 9383 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:32.059424 9133 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:32.072247 6877 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:32.072732 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 59155us
I20260620 19:59:32.075467 6195 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:32.075482 6227 raft_consensus.cc:2889] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260620 19:59:32.080514 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 66064.6us
I20260620 19:59:32.076186 9453 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:32.085628 6271 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.087463 6271 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.058190 9376 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:32.055433 9550 raft_consensus.cc:697] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:32.069222 8864 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260620 19:59:32.079677 6345 connection.cc:256] RPC call timeout handler was delayed by -1.8634549539999625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
I20260620 19:59:32.071259 9475 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:32.098238 6345 connection.cc:256] RPC call timeout handler was delayed by -1.8634549539999625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
W20260620 19:59:32.098567 6345 connection.cc:256] RPC call timeout handler was delayed by -1.8166549540000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20260620 19:59:32.098868 6345 connection.cc:256] RPC call timeout handler was delayed by -1.711354953999944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
W20260620 19:59:32.099148 6345 connection.cc:256] RPC call timeout handler was delayed by -1.6492549539999573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1889s for a response
W20260620 19:59:32.099408 6345 connection.cc:256] RPC call timeout handler was delayed by -1.5043549539999503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.205s for a response
W20260620 19:59:32.101432 6345 connection.cc:256] RPC call timeout handler was delayed by -1.4674549540000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
W20260620 19:59:32.101712 6345 connection.cc:256] RPC call timeout handler was delayed by -1.3792549539999754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
W20260620 19:59:32.102006 6345 connection.cc:256] RPC call timeout handler was delayed by -1.1623549539999658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24300000000000002s for a response
I20260620 19:59:32.077014 9352 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:32.067992 9379 raft_consensus.cc:740] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:32.097214 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:32.075281 9429 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260620 19:59:32.050324 9566 consensus_queue.cc:1048] T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.042s
I20260620 19:59:32.080579 9133 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
W20260620 19:59:32.111030 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 187338us
I20260620 19:59:32.134532 6345 leader_election.cc:304] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:32.094853 9464 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:32.105091 8864 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:32.140489 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 68332.9us
W20260620 19:59:32.142524 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 45387.4us
I20260620 19:59:32.106875 9475 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:32.077214 9408 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.107282 9428 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:32.143046 6877 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:32.141125 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 53750.3us
I20260620 19:59:32.109695 9381 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:32.149511 6345 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
W20260620 19:59:32.131901 9539 log.cc:927] Time spent T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.283s user 0.001s sys 0.000s
I20260620 19:59:32.158166 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.158038 6877 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.102965 9376 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.137049 9319 raft_consensus.cc:2804] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:32.136746 9554 raft_consensus.cc:2749] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.141993 9369 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:32.247117 6345 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:32.148162 8864 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Advancing to term 11
W20260620 19:59:32.247100 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 95344.7us
W20260620 19:59:32.253059 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:32.112528 9386 consensus_queue.cc:1048] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260620 19:59:32.111979 9499 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.266793 6345 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.269709 6345 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.157315 9429 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.271097 6345 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.241801 9546 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:32.244863 9530 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.157107 9464 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
W20260620 19:59:32.266731 9545 log.cc:927] Time spent T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260620 19:59:32.280457 6198 leader_election.cc:304] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.283370 6345 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.169014 9339 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.261123 9369 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:32.165506 9497 consensus_queue.cc:1048] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.118s
W20260620 19:59:32.291260 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 155578us
W20260620 19:59:32.291949 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 126972us
I20260620 19:59:32.158260 9466 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.269074 9358 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.302836 6271 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.240293 9558 raft_consensus.cc:2749] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.282370 9477 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:32.292755 9481 raft_consensus.cc:2749] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:32.253660 9547 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.276856 9569 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:32.279878 9546 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:32.278566 9487 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.243588 9442 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.283443 9319 raft_consensus.cc:697] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:32.302129 9440 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.154709 6377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:59:32.322906 6377 raft_consensus.cc:606] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.323328 6377 raft_consensus.cc:614] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:32.288371 9507 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.295724 9369 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:32.296222 9566 raft_consensus.cc:2749] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.332113 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1783) took 1093 ms. Trace:
I20260620 19:59:32.330572 9394 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:32.333187 6379 rpcz_store.cc:276] 0620 19:59:31.238704 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.265218 (+ 26514us) service_pool.cc:224] Handling call
0620 19:59:31.265509 (+ 291us) raft_consensus.cc:1471] Updating replica for 3 ops
0620 19:59:31.487757 (+222248us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:31.487793 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0620 19:59:31.488454 (+ 661us) log.cc:844] Serialized 78 byte log entry
0620 19:59:31.685631 (+197177us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:31.694910 (+ 9279us) log.cc:844] Serialized 14 byte log entry
0620 19:59:31.695094 (+ 184us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:31.695172 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:32.331801 (+636629us) raft_consensus.cc:1685] finished
0620 19:59:32.331830 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:32.332032 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":835028,"prepare.queue_time_us":12064,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":3179,"raft.queue_time_us":394569,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":616196,"thread_start_us":196717,"threads_started":1,"wal-append.queue_time_us":213137}
I20260620 19:59:32.320505 9430 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:32.328248 9564 log.cc:927] Time spent T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.273s user 0.001s sys 0.000s
I20260620 19:59:32.322027 9546 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:32.320251 9569 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:32.338680 6228 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
W20260620 19:59:32.352217 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:32.274742 9550 consensus_queue.cc:237] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.360733 9394 raft_consensus.cc:697] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:32.364727 9569 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Advancing to term 10
W20260620 19:59:32.338299 9555 log.cc:927] Time spent T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.228s user 0.001s sys 0.001s
I20260620 19:59:32.333958 9351 raft_consensus.cc:2749] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:32.275077 9379 consensus_queue.cc:260] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:32.395160 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 237354us
W20260620 19:59:32.361672 9365 consensus_meta.cc:287] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.635s user 0.005s sys 0.004s
I20260620 19:59:32.404762 6877 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.335101 9428 raft_consensus.cc:697] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:32.361280 9376 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.410900 6877 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.357829 9430 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.321343 9464 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.419310 6877 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:32.412233 9339 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:32.424772 6877 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:32.429315 9568 log.cc:927] Time spent T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
W20260620 19:59:32.429628 9552 log.cc:927] Time spent T bb9f062c40bd4cb68d470ec9b5e63a92 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.258s user 0.001s sys 0.000s
W20260620 19:59:32.435612 9236 consensus_meta.cc:287] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.596s user 0.010s sys 0.000s
I20260620 19:59:32.435768 9577 consensus_queue.cc:1059] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260620 19:59:32.423538 9429 raft_consensus.cc:2804] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260620 19:59:32.440074 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 159702us
I20260620 19:59:32.441386 6298 raft_consensus.cc:2410] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:32.441608 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 869) took 1420 ms. Trace:
I20260620 19:59:32.445068 6911 rpcz_store.cc:276] 0620 19:59:31.020909 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.021150 (+ 241us) service_pool.cc:224] Handling call
0620 19:59:31.044593 (+ 23443us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:31.045006 (+ 413us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:31.045045 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:31.463195 (+418150us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b18003fc5e0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b5796aa6 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b71b3c71 00007fa6b71c35d2 00007fa6b88f5de1 00007fa6b88ba81a
0620 19:59:31.780246 (+317051us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b2c000125b8. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b71b3c71 00007fa6b71c35d2 00007fa6b88f5de1 00007fa6b88ba81a 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22
0620 19:59:31.780743 (+ 497us) log.cc:844] Serialized 26 byte log entry
0620 19:59:31.867484 (+ 86741us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:31.867522 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:31.867587 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:32.441398 (+573811us) raft_consensus.cc:1685] finished
0620 19:59:32.441417 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:32.441554 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2943,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":24082688,"thread_start_us":86390,"threads_started":1,"wal-append.queue_time_us":280898}
I20260620 19:59:32.440464 6877 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.421864 9408 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:32.411535 9365 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.432158 9499 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:32.452858 6877 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.437546 9566 raft_consensus.cc:2749] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.455957 6345 leader_election.cc:304] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:32.454720 9428 consensus_queue.cc:237] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.450776 9394 consensus_queue.cc:237] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.456518 9570 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.456700 9442 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:32.456991 6377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:59:32.454689 9339 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:32.462852 9352 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.454393 6198 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:32.498967 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 44627.1us
W20260620 19:59:32.464665 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:32.456338 9562 log.cc:927] Time spent T 8b0538f91fad4439a7d30adbf2f83b57 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.271s user 0.001s sys 0.001s
I20260620 19:59:32.474911 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:32.422015 9319 consensus_queue.cc:237] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.444739 9429 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:32.496397 6377 raft_consensus.cc:606] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:32.519306 6377 raft_consensus.cc:614] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:32.477025 9466 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:32.473711 9498 raft_consensus.cc:2749] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.502643 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.506088 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:32.480655 9294 consensus_meta.cc:287] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.622s user 0.009s sys 0.001s
I20260620 19:59:32.471194 9566 raft_consensus.cc:2749] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:32.526150 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.528640 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.529968 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.530917 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.533814 6910 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:32.535494 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.503530 9544 raft_consensus.cc:2749] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:32.540141 9365 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.540398 9352 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.543438 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.512168 9430 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:32.526091 9429 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:32.551108 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.465564 9475 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.526257 9487 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:32.553874 6231 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260620 19:59:32.554699 6226 raft_consensus.cc:1240] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:32.476302 9453 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.570430 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.574293 6230 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:32.576081 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:32.582146 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.582274 6904 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
I20260620 19:59:32.550813 8864 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.597429 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:32.621950 6372 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.010s sys 0.012s
I20260620 19:59:32.632853 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.633559 6231 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 8.
I20260620 19:59:32.633906 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:32.634442 6377 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:32.635785 6345 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:32.633150 6372 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.582s user 0.011s sys 0.014s
I20260620 19:59:32.637964 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.637574 6907 raft_consensus.cc:1275] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260620 19:59:32.576865 9487 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.640641 6271 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.645161 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 7 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.642060 6372 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "eb2f228ebe40440d877be6302bba2cd6"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:59:32.646021 6372 raft_consensus.cc:606] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Received request to transfer leadership to TS eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.646353 6372 raft_consensus.cc:614] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:32.646785 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.647881 6271 leader_election.cc:304] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.649336 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.644775 9498 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:32.652208 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.653151 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.657526 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.656857 6908 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260620 19:59:32.669003 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 366797us
I20260620 19:59:32.669157 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.638638 9566 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:32.636958 9578 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:32.654479 9498 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:32.675480 6271 leader_election.cc:304] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.651661 9497 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:32.654896 9408 raft_consensus.cc:2749] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.673166 6196 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:32.679028 6345 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:32.655514 9477 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:32.678524 9133 consensus_meta.cc:287] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.004s
I20260620 19:59:32.709725 9497 raft_consensus.cc:2749] T bb9f062c40bd4cb68d470ec9b5e63a92 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.710871 6271 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:32.658685 9481 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:32.639480 9546 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.655213 9577 consensus_queue.cc:1059] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260620 19:59:32.743923 9429 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:32.718426 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 45333.3us
I20260620 19:59:32.719697 6372 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369"
dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
mode: GRACEFUL
new_leader_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
from {username='slave'} at 127.0.0.1:57168
I20260620 19:59:32.714555 9365 consensus_queue.cc:1059] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.006s
I20260620 19:59:32.720993 6224 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45808 (request call id 2032) took 1622 ms. Trace:
I20260620 19:59:32.727062 9544 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:32.712463 9477 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.727411 9481 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.728175 6909 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 961) took 1160 ms. Trace:
I20260620 19:59:32.728386 9546 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.692101 9566 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.729746 6377 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:32.719594 9498 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:32.733150 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:32.770802 6910 raft_consensus.cc:2393] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 9: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:32.733407 9453 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.733996 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:32.734539 8864 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.703533 9547 consensus_queue.cc:1048] T cc7845798261439d855e526604de104a P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.064s
I20260620 19:59:32.735459 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.715191 9483 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:32.735874 9475 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:32.717572 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.728271 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:32.781680 6305 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:32.749045 6907 raft_consensus.cc:1240] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:32.751648 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:32.788057 9566 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:32.752329 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:32.753002 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:32.755901 6372 raft_consensus.cc:606] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Received request to transfer leadership to TS 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:32.792297 6372 raft_consensus.cc:614] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260620 19:59:32.774832 9464 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:32.758744 6224 rpcz_store.cc:276] 0620 19:59:31.098511 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.100054 (+ 1543us) service_pool.cc:224] Handling call
0620 19:59:31.100437 (+ 383us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:31.664261 (+563824us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:32.720744 (+1056483us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54001414d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:32.720935 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1193517440}
I20260620 19:59:32.764468 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.792847 9464 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.764709 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.793850 6301 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1885) took 1638 ms. Trace:
I20260620 19:59:32.764473 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.765615 6909 rpcz_store.cc:276] 0620 19:59:31.567940 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.568881 (+ 941us) service_pool.cc:224] Handling call
0620 19:59:31.662673 (+ 93792us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:31.663484 (+ 811us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:32.727982 (+1064498us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b5400176bd0. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:32.728115 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159513344}
I20260620 19:59:32.747164 9579 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:32.794162 6306 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:32.794713 9579 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:32.795177 9579 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:32.795357 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 958) took 1564 ms. Trace:
I20260620 19:59:32.767740 6345 leader_election.cc:304] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.795723 6906 rpcz_store.cc:276] 0620 19:59:31.230946 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.253597 (+ 22651us) service_pool.cc:224] Handling call
0620 19:59:31.253972 (+ 375us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:31.610762 (+356790us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:32.795119 (+1184357us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400786ed4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:32.795301 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":819078912}
I20260620 19:59:32.796631 6345 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:32.769534 6198 leader_election.cc:304] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:32.772296 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:32.773541 9510 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:32.771301 9577 raft_consensus.cc:3055] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Stepping down as leader of term 9
I20260620 19:59:32.716706 9133 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.711611 9590 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260620 19:59:32.803025 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.779875 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.734692 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.761365 9544 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:32.794126 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:32.775280 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:32.704589 9578 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.799789 6345 leader_election.cc:304] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.794271 6301 rpcz_store.cc:276] 0620 19:59:31.155163 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.155913 (+ 750us) service_pool.cc:224] Handling call
0620 19:59:31.156136 (+ 223us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:31.523205 (+367069us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:32.793660 (+1270455us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54003a6954. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:32.793800 (+ 140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":842708352}
I20260620 19:59:32.776360 9507 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:32.813401 6908 raft_consensus.cc:2368] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8.
I20260620 19:59:32.813242 6303 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:32.839298 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:32.853794 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:32.858196 9586 log.cc:927] Time spent T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260620 19:59:32.858440 6908 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:32.861857 9133 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.814668 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:32.863997 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:32.864027 6196 leader_election.cc:400] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8.
I20260620 19:59:32.864202 6304 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:32.864668 6196 leader_election.cc:403] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260620 19:59:32.864715 6907 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:32.813685 6300 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:32.805768 6306 raft_consensus.cc:2410] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260620 19:59:32.865486 6299 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Advancing to term 11
I20260620 19:59:32.839569 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.868896 6307 raft_consensus.cc:3060] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:32.868686 6232 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:32.800745 9559 raft_consensus.cc:2749] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.814523 9544 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:32.801043 9577 raft_consensus.cc:740] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:32.890259 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.837810 9487 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:32.873200 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:32.872597 9596 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:32.803439 8864 raft_consensus.cc:2804] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:32.862982 9507 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.839390 9369 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.905143 6906 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
W20260620 19:59:32.906080 6874 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.244.130:42141 timed out after 1.526s (SENT) [suppressed 21 similar messages]
I20260620 19:59:32.906625 6304 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 10: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:32.906744 6299 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 11.
I20260620 19:59:32.876519 9464 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:32.799160 9483 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: NON_VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: true } }
I20260620 19:59:32.916038 6305 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:32.918316 8864 raft_consensus.cc:697] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:32.919557 9429 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:32.919579 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 119828us
I20260620 19:59:32.942786 6345 leader_election.cc:304] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.909495 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:32.928588 6300 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 10.
I20260620 19:59:32.935967 6307 raft_consensus.cc:2468] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:32.907038 9547 raft_consensus.cc:2749] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:32.945325 6345 leader_election.cc:304] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.840047 9510 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.950134 6345 leader_election.cc:304] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:32.950917 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 44866.8us
I20260620 19:59:32.936781 9592 raft_consensus.cc:3060] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:32.954262 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:32.906896 9550 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260620 19:59:32.955708 6345 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:32.956734 9603 raft_consensus.cc:2749] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.957470 8864 consensus_queue.cc:237] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.959834 6302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 783) took 1021 ms. Trace:
I20260620 19:59:32.945164 9365 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:32.966183 9383 consensus_meta.cc:287] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.875s user 0.008s sys 0.000s
I20260620 19:59:32.959851 9549 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:32.960934 9547 raft_consensus.cc:2749] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:32.974540 9365 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:32.976279 9460 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:32.972321 9487 raft_consensus.cc:2749] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.976939 9365 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:32.978051 9483 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:32.986040 6906 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:32.971690 6302 rpcz_store.cc:276] 0620 19:59:31.938421 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.938568 (+ 147us) service_pool.cc:224] Handling call
0620 19:59:31.938892 (+ 324us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:31.990855 (+ 51963us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:31.990899 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:31.990955 (+ 56us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:31.991008 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:32.959618 (+968610us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b54003a6454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:32.959665 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:32.959777 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118972672}
I20260620 19:59:32.986558 9383 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:32.953346 9429 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:32.949221 9559 raft_consensus.cc:2749] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:32.993180 6302 raft_consensus.cc:1240] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:33.005636 6906 raft_consensus.cc:1240] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:32.994875 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:32.997148 6874 leader_election.cc:304] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.998910 6195 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:33.007804 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.008322 6375 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 10
I20260620 19:59:33.009270 6195 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:32.966252 9498 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.011425 6372 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:33.012668 9498 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:33.013643 9556 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.896s user 0.003s sys 0.000s
I20260620 19:59:32.999580 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.000999 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:33.001657 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:33.001892 9547 consensus_queue.cc:1059] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:59:33.002935 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:33.021927 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:33.027138 6195 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:33.031826 6228 raft_consensus.cc:2393] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 8: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:33.032024 6195 leader_election.cc:304] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:33.029675 6874 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:33.032816 6345 leader_election.cc:304] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:33.035068 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:33.035805 6227 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 10: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:33.033638 6874 leader_election.cc:304] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
W20260620 19:59:33.037199 6877 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:33.038528 6345 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:33.006139 9365 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:33.045543 6375 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 10.
I20260620 19:59:33.006883 9477 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.048251 6195 leader_election.cc:304] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:33.047981 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:33.007037 9549 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:32.968675 9577 consensus_queue.cc:260] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.016012 9483 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260620 19:59:33.047539 9493 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:33.057179 9493 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:33.057551 9493 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:33.030954 9430 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:32.943650 9596 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.047683 9354 consensus_queue.cc:1059] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.013s
I20260620 19:59:33.040644 9475 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:33.032104 9464 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:33.047947 9406 raft_consensus.cc:2804] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:33.050531 9428 raft_consensus.cc:2804] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 11 FOLLOWER]: Leader election won for term 11
I20260620 19:59:33.067520 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.014241 9429 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:33.062525 9569 consensus_meta.cc:287] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.652s user 0.011s sys 0.000s
I20260620 19:59:33.030166 9498 consensus_queue.cc:1059] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260620 19:59:33.047839 9481 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.060879 9499 raft_consensus.cc:2804] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader election won for term 10
I20260620 19:59:33.111794 9499 raft_consensus.cc:697] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [term 10 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:33.053767 9549 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:33.112644 9499 consensus_queue.cc:237] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:33.061975 9559 raft_consensus.cc:2749] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:32.982399 9579 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:33.062451 9477 raft_consensus.cc:2749] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:33.096472 9585 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:33.073549 9453 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:33.104257 9429 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:33.105923 9510 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:33.106585 9493 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.107724 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:33.141875 6305 raft_consensus.cc:2368] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 9. Current term is 10.
I20260620 19:59:33.070362 9475 raft_consensus.cc:493] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:33.053707 9365 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:33.116232 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 84280.1us
I20260620 19:59:33.104139 9498 raft_consensus.cc:3055] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Stepping down as leader of term 4
W20260620 19:59:33.119081 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 80608us
I20260620 19:59:33.096841 9406 raft_consensus.cc:697] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:33.113595 9428 raft_consensus.cc:697] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [term 11 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:33.158802 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:33.160737 6345 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:33.160995 6305 raft_consensus.cc:2368] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 10. Current term is 11.
I20260620 19:59:33.152366 6273 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:33.165282 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.166651 6306 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 8.
I20260620 19:59:33.098053 9569 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.148636 9507 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:33.150800 9475 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:33.170264 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:33.178150 6902 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:33.178988 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:33.151947 9498 raft_consensus.cc:740] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:33.183758 5074 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260620 19:59:33.183828 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.184291 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.184585 6227 raft_consensus.cc:2393] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:33.185391 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.129761 9585 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.163827 9525 raft_consensus.cc:3055] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:33.159368 9369 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.139896 9453 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.200898 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.214625 6911 raft_consensus.cc:2393] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 10: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:33.204273 9585 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.199873 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.166757 9367 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader election won for term 10
W20260620 19:59:33.206151 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:33.206085 9570 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:33.207068 6874 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:33.152269 9544 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.572656 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:33.221993 6298 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 778) took 2040 ms (client timeout 1526 ms). Trace:
I20260620 19:59:33.610201 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.304080 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfeb1fb4ea6746dbaf0509f4097e00ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:33.633574 6227 raft_consensus.cc:2393] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 10: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:33.633679 6910 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:33.572427 6344 leader_election.cc:400] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 9. Current term is 10.
I20260620 19:59:33.634277 6344 leader_election.cc:403] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:59:33.543920 6875 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:33.631126 9429 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:33.302358 9406 consensus_queue.cc:237] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.636492 9429 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.464344 9538 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:33.638559 9538 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:33.638875 9429 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:33.640430 9538 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:33.540005 9440 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:33.641546 9549 consensus_meta.cc:287] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.528s user 0.008s sys 0.000s
I20260620 19:59:33.641304 9440 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.511438 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:33.522909 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.644733 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:33.220850 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 890) took 1313 ms. Trace:
I20260620 19:59:33.645313 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0153cc2373244ab0b2574d1de595c4a4" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:33.646909 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 103061us
I20260620 19:59:33.645649 6904 rpcz_store.cc:276] 0620 19:59:31.907683 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.912434 (+ 4751us) service_pool.cc:224] Handling call
0620 19:59:31.912703 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:32.582945 (+670242us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:33.220637 (+637692us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b540054fbd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:33.220798 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1539158784}
I20260620 19:59:33.409977 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2105) took 1316 ms. Trace:
I20260620 19:59:33.651875 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.652580 6377 raft_consensus.cc:3060] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 10
I20260620 19:59:33.654793 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.358688 5074 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260620 19:59:33.658975 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
W20260620 19:59:33.645740 6298 rpcz_store.cc:269] 0620 19:59:31.181122 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.214155 (+ 33033us) service_pool.cc:224] Handling call
0620 19:59:33.221814 (+2007659us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001cdc54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:33.221933 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2777350528}
I20260620 19:59:33.659529 6380 raft_consensus.cc:3060] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Advancing to term 11
I20260620 19:59:33.647928 6875 leader_election.cc:304] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:33.368083 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:33.645629 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.660856 6227 raft_consensus.cc:2393] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 10: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
W20260620 19:59:33.630734 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 294678us
W20260620 19:59:33.413228 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:33.650185 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:33.296800 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 94904.1us
I20260620 19:59:33.664038 6345 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:33.665467 6345 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:33.666692 6877 leader_election.cc:304] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:33.512240 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2092) took 1958 ms. Trace:
I20260620 19:59:33.442194 6907 raft_consensus.cc:1240] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:33.651485 6374 rpcz_store.cc:276] 0620 19:59:32.093638 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:32.093760 (+ 122us) service_pool.cc:224] Handling call
0620 19:59:32.094093 (+ 333us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:32.094462 (+ 369us) raft_consensus.cc:1534] Early marking committed up to index 5
0620 19:59:32.131739 (+ 37277us) log.cc:844] Serialized 14 byte log entry
0620 19:59:32.292012 (+160273us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:32.296950 (+ 4938us) log.cc:844] Serialized 52 byte log entry
0620 19:59:32.297100 (+ 150us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:33.409293 (+1112193us) log.cc:844] Serialized 14 byte log entry
0620 19:59:33.409550 (+ 257us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:33.409650 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:33.409707 (+ 57us) raft_consensus.cc:1685] finished
0620 19:59:33.409725 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:33.409904 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2253530,"prepare.queue_time_us":1172988,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"raft.queue_time_us":1949,"raft.run_cpu_time_us":431,"raft.run_wall_time_us":23531,"thread_start_us":46680,"threads_started":1,"wal-append.queue_time_us":272840}
I20260620 19:59:33.212250 9607 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:33.683673 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:33.691745 6902 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:33.668242 6378 rpcz_store.cc:276] 0620 19:59:31.554034 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.554165 (+ 131us) service_pool.cc:224] Handling call
0620 19:59:31.582573 (+ 28408us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:31.874105 (+291532us) raft_consensus.cc:1534] Early marking committed up to index 5
0620 19:59:31.955537 (+ 81432us) log.cc:844] Serialized 14 byte log entry
0620 19:59:32.348681 (+393144us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:32.377633 (+ 28952us) log.cc:844] Serialized 52 byte log entry
0620 19:59:32.377841 (+ 208us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:33.410849 (+1033008us) log.cc:844] Serialized 14 byte log entry
0620 19:59:33.511897 (+101048us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:33.511991 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:33.512034 (+ 43us) raft_consensus.cc:1685] finished
0620 19:59:33.512061 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:33.512186 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1148430,"prepare.queue_time_us":1122929,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":305781,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":135470,"thread_start_us":632167,"threads_started":2,"wal-append.queue_time_us":370188}
I20260620 19:59:33.685671 6910 raft_consensus.cc:2468] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:33.688481 9406 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260620 19:59:33.693073 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:33.694170 6229 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:33.692329 6345 leader_election.cc:304] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:33.699486 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:33.220243 9602 log.cc:927] Time spent T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
W20260620 19:59:33.507368 9592 consensus_meta.cc:287] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.536s user 0.009s sys 0.001s
I20260620 19:59:33.695365 6911 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260620 19:59:33.695907 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
W20260620 19:59:33.694407 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 125391us
I20260620 19:59:33.631327 9481 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:33.572955 9560 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:33.510737 9236 raft_consensus.cc:3055] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:33.707319 6228 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
W20260620 19:59:33.708613 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:33.641947 9549 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:33.569221 9417 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:33.718179 6377 raft_consensus.cc:2468] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 10.
I20260620 19:59:33.381245 9523 raft_consensus.cc:3055] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:33.372658 9569 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:33.353708 9453 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:33.210430 9365 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:33.528884 9383 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:33.477900 9477 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:33.212682 9525 raft_consensus.cc:740] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:33.331231 9579 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:33.209669 9578 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:33.379629 9596 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.332119 9498 consensus_queue.cc:260] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.773651 9523 raft_consensus.cc:740] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:33.725639 6344 leader_election.cc:400] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 10. Current term is 11.
I20260620 19:59:33.832927 6344 leader_election.cc:403] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260620 19:59:33.727818 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:33.833557 6374 raft_consensus.cc:2468] T 97eee360f90e4475858b88d88f8efc5c P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:33.834393 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.834908 6374 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:33.689399 9358 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:33.658856 9510 raft_consensus.cc:2749] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:33.658634 9429 raft_consensus.cc:2749] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260620 19:59:33.737974 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.844064 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.844481 6378 raft_consensus.cc:2393] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 10: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:33.504565 9487 raft_consensus.cc:3055] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:33.747241 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:33.747450 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:33.683223 9367 raft_consensus.cc:697] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [term 10 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:33.850883 6195 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:33.568747 9548 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:33.690920 9566 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:33.691305 9294 raft_consensus.cc:2749] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:33.704487 9538 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:33.771229 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.724030 9592 raft_consensus.cc:2749] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325) with higher term. Message: Invalid argument: T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8.
I20260620 19:59:33.783998 6299 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260620 19:59:33.787262 6379 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:33.690807 9547 consensus_queue.cc:1059] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.083s
I20260620 19:59:33.885855 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:33.799310 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 107024us
I20260620 19:59:33.785480 9406 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:33.737884 9236 raft_consensus.cc:740] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:33.303988 9428 consensus_queue.cc:237] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:33.706374 9369 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260620 19:59:33.893368 6298 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:33.658707 9570 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.331828 9493 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.824430 9440 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:33.826767 9625 raft_consensus.cc:2749] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 9. Current term is 10.
I20260620 19:59:33.852420 9603 raft_consensus.cc:2749] T a56f959cee394e398810b9170b9d24be P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260620 19:59:33.823292 9477 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.903550 6231 raft_consensus.cc:2468] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:33.830271 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
W20260620 19:59:33.824445 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 162248us
I20260620 19:59:33.737654 9481 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.884042 9429 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:33.839237 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 115319us
I20260620 19:59:33.841483 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:33.823405 9602 log.cc:927] Time spent T a5739ffa3b914095862145adf0c084fb P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260620 19:59:33.886590 9499 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:33.840936 6380 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 11.
I20260620 19:59:33.844720 6225 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:33.915730 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:33.849440 6346 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:33.554266 9233 consensus_queue.cc:1059] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.167s
I20260620 19:59:33.917979 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:33.852789 6273 leader_election.cc:304] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:33.918874 6198 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:33.919276 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 66552.5us
I20260620 19:59:33.854460 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2106) took 1707 ms. Trace:
I20260620 19:59:33.858264 6223 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260620 19:59:33.858803 6374 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 10.
I20260620 19:59:33.881839 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:33.925428 6377 raft_consensus.cc:2410] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260620 19:59:33.925453 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:33.923658 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:33.926324 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:33.926890 6298 raft_consensus.cc:2393] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 in current term 5: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:33.927078 6377 raft_consensus.cc:2410] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb2f228ebe40440d877be6302bba2cd6 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260620 19:59:33.841310 9376 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:33.841749 9578 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:33.890681 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:33.890084 6345 leader_election.cc:304] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:33.717067 9607 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.898762 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:33.846809 9487 raft_consensus.cc:740] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:33.904531 6230 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:33.912624 6196 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:33.857039 9558 raft_consensus.cc:2749] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:33.916105 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:33.917068 9236 consensus_queue.cc:260] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.917675 9549 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:33.776819 9560 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.852859 6301 raft_consensus.cc:1275] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260620 19:59:33.920879 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:33.844153 9559 consensus_queue.cc:1175] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" responder_term: 4 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260620 19:59:33.924775 9592 raft_consensus.cc:2749] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260620 19:59:33.930574 6198 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:33.883880 9566 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.931114 6375 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:33.928429 9383 raft_consensus.cc:2804] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260620 19:59:33.930524 6303 raft_consensus.cc:3060] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 9
I20260620 19:59:33.918485 9610 raft_consensus.cc:2749] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 39662d01be7a44f682d160227f457661 (127.4.244.130:42141) with higher term. Message: Invalid argument: T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 10. Current term is 11.
I20260620 19:59:33.929281 9603 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:33.922750 9475 consensus_meta.cc:287] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.700s user 0.006s sys 0.004s
I20260620 19:59:33.936334 9585 raft_consensus.cc:3055] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Stepping down as leader of term 9
W20260620 19:59:33.936583 9614 log.cc:927] Time spent T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20260620 19:59:33.857311 9548 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:33.937947 9620 log.cc:927] Time spent T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260620 19:59:33.938226 9367 consensus_queue.cc:237] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:33.940968 6377 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:33.946743 9544 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.919734 9429 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.919598 9499 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:33.897279 9406 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.948550 6298 raft_consensus.cc:1275] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:33.948856 9625 raft_consensus.cc:2749] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:33.926349 9440 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:33.887487 9525 consensus_queue.cc:260] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:33.957172 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 44615.8us
I20260620 19:59:33.921315 6376 rpcz_store.cc:276] 0620 19:59:32.147463 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:32.148336 (+ 873us) service_pool.cc:224] Handling call
0620 19:59:32.150963 (+ 2627us) raft_consensus.cc:1471] Updating replica for 2 ops
0620 19:59:32.162892 (+ 11929us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:32.162922 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0620 19:59:32.262581 (+ 99659us) log.cc:844] Serialized 52 byte log entry
0620 19:59:32.407903 (+145322us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:33.726371 (+1318468us) log.cc:844] Serialized 14 byte log entry
0620 19:59:33.853471 (+127100us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:33.854232 (+ 761us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:33.854275 (+ 43us) raft_consensus.cc:1685] finished
0620 19:59:33.854287 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:33.854400 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1003022,"prepare.queue_time_us":1028368,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":64779,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":83566,"thread_start_us":533648,"threads_started":3,"wal-append.queue_time_us":197968,"wal-append.run_cpu_time_us":1871,"wal-append.run_wall_time_us":1015512}
I20260620 19:59:33.959884 6875 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
W20260620 19:59:34.008555 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 92506.3us
W20260620 19:59:34.053712 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 93874.3us
I20260620 19:59:33.964298 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 896) took 1203 ms. Trace:
I20260620 19:59:33.966763 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 903) took 1084 ms. Trace:
I20260620 19:59:34.054795 6906 rpcz_store.cc:276] 0620 19:59:32.882116 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.059941 (+177825us) service_pool.cc:224] Handling call
0620 19:59:33.060208 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:33.113939 (+ 53731us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:33.113967 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:33.114007 (+ 40us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:33.114051 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:33.966454 (+852403us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b540044e3d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:33.966511 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:33.966693 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123157760}
I20260620 19:59:33.973209 6224 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260620 19:59:33.932500 6345 leader_election.cc:304] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:33.931201 6378 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 7.
I20260620 19:59:34.023689 6230 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:34.025668 6301 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260620 19:59:33.927877 9388 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
W20260620 19:59:34.027405 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 96894.4us
I20260620 19:59:33.961417 9236 raft_consensus.cc:3060] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:34.066149 9477 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:33.928006 9550 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260620 19:59:34.066505 9481 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.067751 9560 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.068948 6271 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:34.049636 9499 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:34.073226 9570 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:34.074232 9583 consensus_queue.cc:1059] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:34.074939 9583 consensus_queue.cc:1243] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [LEADER]: Peer eb2f228ebe40440d877be6302bba2cd6 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260620 19:59:34.075415 9583 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:34.019268 9623 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.069s
I20260620 19:59:34.035159 9585 raft_consensus.cc:740] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:34.046626 6306 raft_consensus.cc:2410] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:34.039111 6375 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 8.
I20260620 19:59:34.048676 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:34.077677 6908 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:33.955420 9578 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.078740 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:34.079975 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:34.054736 6907 rpcz_store.cc:276] 0620 19:59:32.761155 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:32.881282 (+120127us) service_pool.cc:224] Handling call
0620 19:59:33.441939 (+560657us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:33.670104 (+228165us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:33.964064 (+293960us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b54002a3350. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:33.964235 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1288621312}
I20260620 19:59:34.080292 6305 raft_consensus.cc:1275] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260620 19:59:34.080693 6305 pending_rounds.cc:85] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 7
I20260620 19:59:34.046070 9475 raft_consensus.cc:515] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.080902 6305 pending_rounds.cc:107] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260620 19:59:34.081187 6305 raft_consensus.cc:2889] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:34.081437 6271 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters:
I20260620 19:59:34.084872 6302 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260620 19:59:34.085233 6302 pending_rounds.cc:85] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Aborting all ops after (but not including) 7
I20260620 19:59:34.085449 6302 pending_rounds.cc:107] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260620 19:59:34.085690 6302 raft_consensus.cc:2889] T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:34.040084 9603 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.087488 9560 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:34.087913 9560 raft_consensus.cc:493] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:34.088249 9560 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:34.088536 9603 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.030506 8829 consensus_queue.cc:1048] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260620 19:59:34.035475 9493 raft_consensus.cc:2749] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:33.968441 9496 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260620 19:59:34.095219 6908 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:34.095789 6909 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260620 19:59:34.035455 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:34.075845 9583 raft_consensus.cc:697] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [term 9 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:34.019637 9369 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:34.103379 9583 consensus_queue.cc:237] T 098d4ada753442e1899d391a485095aa P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:34.077275 9585 consensus_queue.cc:260] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:33.956422 9376 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:34.107514 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 175067us
I20260620 19:59:34.107760 6904 raft_consensus.cc:2368] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9.
I20260620 19:59:34.080166 9578 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:34.098341 6909 raft_consensus.cc:1240] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260620 19:59:34.113179 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 60282.7us
I20260620 19:59:34.115002 6877 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:34.074457 9607 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.141765 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:34.143667 6909 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:34.148169 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:34.117283 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 58613us
I20260620 19:59:34.103401 9369 raft_consensus.cc:493] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:34.020457 9383 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260620 19:59:34.117470 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 57282.2us
W20260620 19:59:34.157059 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:34.159318 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:34.138011 9549 raft_consensus.cc:2804] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:34.116806 9612 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:34.138669 9525 raft_consensus.cc:3060] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:34.168541 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:34.163244 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:34.170749 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:34.163914 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:34.171528 6378 raft_consensus.cc:2468] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:34.172161 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:34.146098 9585 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:34.167228 6902 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:34.177511 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:34.177974 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:34.178517 6224 raft_consensus.cc:2368] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 8. Current term is 9.
I20260620 19:59:34.179339 6271 leader_election.cc:304] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:34.160910 6303 raft_consensus.cc:2468] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:34.163771 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:34.140112 9496 raft_consensus.cc:697] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:34.111649 9294 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:34.152055 6345 leader_election.cc:304] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:34.174078 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:34.185593 6906 raft_consensus.cc:2393] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 8: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:34.169301 6223 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:34.159530 9369 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:34.139521 9537 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20260620 19:59:34.098356 9560 raft_consensus.cc:515] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.051445 9627 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:34.201678 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:34.173643 9578 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.137687 9375 consensus_queue.cc:1059] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260620 19:59:34.186259 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:34.136860 9233 consensus_queue.cc:1059] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260620 19:59:34.149354 9570 raft_consensus.cc:2749] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:34.160303 9383 raft_consensus.cc:3060] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:34.187512 6902 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:34.058099 9592 consensus_queue.cc:1048] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20260620 19:59:34.183763 9236 raft_consensus.cc:2804] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader election won for term 5
W20260620 19:59:34.222729 9628 log.cc:927] Time spent T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20260620 19:59:34.222914 9236 raft_consensus.cc:697] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:34.223994 9236 consensus_queue.cc:237] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.193915 9376 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.179431 9464 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:34.226984 9464 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.227054 9590 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:34.177474 9612 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.229421 9612 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:34.196223 9388 consensus_queue.cc:1059] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260620 19:59:34.230504 9388 raft_consensus.cc:3055] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Stepping down as leader of term 9
I20260620 19:59:34.230929 9388 raft_consensus.cc:740] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:34.222922 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:34.231354 9375 raft_consensus.cc:3055] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260620 19:59:34.232082 6381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2107) took 2073 ms. Trace:
I20260620 19:59:34.231724 9388 consensus_queue.cc:260] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.199849 9379 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:34.232398 9496 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.232460 6381 rpcz_store.cc:276] 0620 19:59:32.158385 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:32.158677 (+ 292us) service_pool.cc:224] Handling call
0620 19:59:32.158962 (+ 285us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:32.159281 (+ 319us) raft_consensus.cc:1534] Early marking committed up to index 3
0620 19:59:33.384571 (+1225290us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b1800350c80. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b574340a 00007fa6bee885d9 00007fa6b57cb240 00007fa6b57c5ce4 00007fa6b57c5c55 00007fa6bc2b4247 00007fa6b88c4d87 00007fa6b88d449f 00007fa6b88d4418
0620 19:59:33.384888 (+ 317us) log.cc:844] Serialized 14 byte log entry
0620 19:59:33.629798 (+244910us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:33.647404 (+ 17606us) log.cc:844] Serialized 26 byte log entry
0620 19:59:33.647536 (+ 132us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:33.647568 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:33.648131 (+ 563us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:34.151912 (+503781us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b1800350c80. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b57c8bf3 00007fa6b57cafc1 00007fa6b57caf79 00007fa6b57caf09 00007fa6b57caed1 00007fa6b57ca1cd 0000000000389354 0000000000389199 00007fa6b57ac03a
0620 19:59:34.231880 (+ 79968us) raft_consensus.cc:1685] finished
0620 19:59:34.231900 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:34.232026 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":186163,"prepare.queue_time_us":1247689,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":74819,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":149076,"spinlock_wait_cycles":18831616,"thread_start_us":171566,"threads_started":1,"wal-append.queue_time_us":275467}
I20260620 19:59:34.173257 9544 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260620 19:59:34.201053 9358 raft_consensus.cc:2749] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:34.181425 6195 leader_election.cc:304] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:34.181661 9549 raft_consensus.cc:697] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Becoming Leader. State: Replica: 3a9a1e04ea8f425f8bcec4439ee5ebf9, State: Running, Role: LEADER
I20260620 19:59:34.205328 6905 raft_consensus.cc:2410] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260620 19:59:34.207916 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 55927.2us
W20260620 19:59:34.248553 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 67227.7us
I20260620 19:59:34.248972 6877 leader_election.cc:304] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:34.186435 9240 consensus_queue.cc:1059] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
I20260620 19:59:34.211078 9365 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:34.214623 9625 raft_consensus.cc:2804] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260620 19:59:34.235194 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:34.256496 6902 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:34.106930 9487 consensus_queue.cc:260] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.063392 9440 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.240182 6273 leader_election.cc:304] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:34.255148 6345 leader_election.cc:400] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 8. Current term is 9.
I20260620 19:59:34.260180 6345 leader_election.cc:403] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260620 19:59:34.214828 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1799) took 1701 ms. Trace:
W20260620 19:59:34.258252 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:34.261629 6373 rpcz_store.cc:276] 0620 19:59:32.513336 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:32.514219 (+ 883us) service_pool.cc:224] Handling call
0620 19:59:32.514581 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:32.514834 (+ 253us) raft_consensus.cc:1534] Early marking committed up to index 4
0620 19:59:33.772166 (+1257332us) log.cc:844] Serialized 14 byte log entry
0620 19:59:34.184823 (+412657us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:34.184872 (+ 49us) raft_consensus.cc:1645] Marking committed up to 4
0620 19:59:34.184915 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:34.184999 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:34.214760 (+ 29761us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":948647,"prepare.queue_time_us":921554,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":63911,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":6125,"thread_start_us":714605,"threads_started":3,"wal-append.queue_time_us":334767}
I20260620 19:59:34.273089 6374 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 10.
I20260620 19:59:34.231741 9375 raft_consensus.cc:740] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:34.253528 9549 consensus_queue.cc:237] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.255684 9365 raft_consensus.cc:493] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:34.253839 9466 raft_consensus.cc:493] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:34.258430 9570 raft_consensus.cc:2804] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:34.255614 9262 raft_consensus.cc:2804] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:34.213292 9499 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:34.288089 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 378813us
I20260620 19:59:34.215106 9578 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.257570 6902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:34.260039 9625 raft_consensus.cc:493] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.277004 6345 leader_election.cc:304] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:34.282172 9358 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:34.285521 9233 raft_consensus.cc:2804] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader election won for term 10
I20260620 19:59:34.341621 9570 raft_consensus.cc:493] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:34.361032 9570 raft_consensus.cc:3060] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:34.347944 9262 raft_consensus.cc:697] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:34.205443 9627 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.338711 9365 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:34.363227 9625 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:34.386413 6307 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
W20260620 19:59:34.386344 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 109396us
I20260620 19:59:34.387264 6226 raft_consensus.cc:2393] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 9: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:34.387411 6345 leader_election.cc:304] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:34.387766 9233 raft_consensus.cc:697] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 10 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:34.363554 9262 consensus_queue.cc:237] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.389109 9607 raft_consensus.cc:2749] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260620 19:59:34.386292 9631 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:34.389892 9518 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:34.402935 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:34.390118 9460 raft_consensus.cc:3055] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:34.366766 9481 raft_consensus.cc:493] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:34.433261 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 100916us
I20260620 19:59:34.371562 9624 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.366606 9630 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.342456 9466 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.454063 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:34.507608 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:34.389889 9631 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.512996 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:34.514953 6376 raft_consensus.cc:1240] T cfeb1fb4ea6746dbaf0509f4097e00ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
W20260620 19:59:34.517168 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 65701.6us
I20260620 19:59:34.420153 9460 raft_consensus.cc:740] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:34.388628 9233 consensus_queue.cc:237] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:34.610771 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:34.616709 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:34.451436 9375 consensus_queue.cc:260] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.619925 9375 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:34.408685 9518 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.523682 9560 leader_election.cc:290] T 758c2974c107466bbcc6c41d00de7b18 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.523123 9632 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:34.459604 9481 raft_consensus.cc:515] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:34.616351 9417 raft_consensus.cc:493] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:34.658309 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 134557us
W20260620 19:59:34.668138 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:34.637173 9648 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.508752 9624 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.507160 9630 raft_consensus.cc:515] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.648038 9577 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
W20260620 19:59:34.701582 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 69139.4us
I20260620 19:59:34.704000 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:34.704730 6298 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:34.674494 9440 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:34.671110 9417 raft_consensus.cc:515] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.666750 9655 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.648274 9632 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.689764 9625 raft_consensus.cc:515] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.657665 9460 consensus_queue.cc:260] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:34.769029 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:34.733714 9538 consensus_meta.cc:287] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.843s user 0.011s sys 0.000s
I20260620 19:59:34.693948 9648 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.893119 9417 leader_election.cc:290] T a56f959cee394e398810b9170b9d24be P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:34.716275 9624 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:34.720306 9499 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:34.895332 9369 consensus_meta.cc:287] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.702s user 0.010s sys 0.000s
I20260620 19:59:34.895926 9507 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:34.897364 6379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 2109) took 1895 ms (client timeout 1553 ms). Trace:
W20260620 19:59:34.909252 6379 rpcz_store.cc:269] 0620 19:59:33.002287 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.002680 (+ 393us) service_pool.cc:224] Handling call
0620 19:59:34.897165 (+1894485us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b540028dcd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:34.897274 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1760504576}
I20260620 19:59:34.783133 9440 raft_consensus.cc:493] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.889047 9354 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:34.899381 9294 consensus_meta.cc:287] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.712s user 0.008s sys 0.000s
I20260620 19:59:34.902120 6298 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
W20260620 19:59:34.884832 9585 consensus_meta.cc:287] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.703s user 0.009s sys 0.000s
I20260620 19:59:34.737612 9578 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:34.897671 9365 consensus_meta.cc:287] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.503s user 0.004s sys 0.004s
I20260620 19:59:34.740383 9630 leader_election.cc:290] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.915640 9624 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:34.927393 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 973) took 1314 ms. Trace:
I20260620 19:59:34.927525 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 641) took 1038 ms. Trace:
I20260620 19:59:34.927803 6904 rpcz_store.cc:276] 0620 19:59:33.613275 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.649888 (+ 36613us) service_pool.cc:224] Handling call
0620 19:59:34.927201 (+1277313us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400587854. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:34.927339 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1023266560}
I20260620 19:59:34.910674 9369 raft_consensus.cc:515] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.884160 9655 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.921162 9354 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:34.901631 9538 raft_consensus.cc:2749] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:34.927906 6374 rpcz_store.cc:276] 0620 19:59:33.889516 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.925258 (+ 35742us) service_pool.cc:224] Handling call
0620 19:59:34.927387 (+1002129us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b5400341554. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:34.927473 (+ 86us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":766082304}
I20260620 19:59:34.739655 9659 raft_consensus.cc:493] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:34.905670 9460 raft_consensus.cc:3060] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:34.915951 9417 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:34.941948 9652 log.cc:927] Time spent T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.224s user 0.001s sys 0.001s
I20260620 19:59:34.921989 9294 raft_consensus.cc:2749] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260620 19:59:34.933913 9660 log.cc:927] Time spent T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
W20260620 19:59:34.946036 9615 log.cc:927] Time spent T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 1.107s user 0.000s sys 0.010s
I20260620 19:59:34.935832 9365 raft_consensus.cc:515] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.948652 6374 raft_consensus.cc:1275] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260620 19:59:34.950259 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2133) took 1054 ms. Trace:
I20260620 19:59:34.922719 9507 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:34.951936 9417 raft_consensus.cc:515] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } attrs { replace: true } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.958921 6908 raft_consensus.cc:1240] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:34.960642 6379 rpcz_store.cc:276] 0620 19:59:33.896034 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.937408 (+1041374us) service_pool.cc:224] Handling call
0620 19:59:34.949714 (+ 12306us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:34.949881 (+ 167us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:34.949908 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:34.949947 (+ 39us) raft_consensus.cc:1645] Marking committed up to 3
0620 19:59:34.949980 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:34.950052 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:34.950182 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:34.960371 6374 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: []
I20260620 19:59:34.923259 9440 raft_consensus.cc:515] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:34.952028 9646 log.cc:927] Time spent T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260620 19:59:34.946653 9664 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
I20260620 19:59:34.938056 9624 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:34.953006 9481 leader_election.cc:290] T bb9f062c40bd4cb68d470ec9b5e63a92 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:34.944216 9631 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:34.986337 9525 consensus_meta.cc:287] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.810s user 0.011s sys 0.000s
I20260620 19:59:35.003892 9294 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.004644 9294 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.005705 6874 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:35.007530 9578 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.008150 9578 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:35.009584 6375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46186 (request call id 638) took 1811 ms (client timeout 2094 ms). Trace:
I20260620 19:59:35.010699 6380 raft_consensus.cc:1240] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 7. Current term is 8. Ops: []
W20260620 19:59:35.010005 6375 rpcz_store.cc:269] 0620 19:59:33.198429 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.199613 (+ 1184us) service_pool.cc:224] Handling call
0620 19:59:35.009408 (+1809795us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b5400065754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.009522 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":646603520}
I20260620 19:59:35.013326 6874 leader_election.cc:304] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:35.010164 9655 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.015048 9655 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.015532 9655 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:35.059360 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:35.078974 6307 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52348 (request call id 1896) took 2085 ms (client timeout 2008 ms). Trace:
W20260620 19:59:35.087232 6307 rpcz_store.cc:269] 0620 19:59:32.993760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:32.993905 (+ 145us) service_pool.cc:224] Handling call
0620 19:59:35.078761 (+2084856us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001cd754. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.078888 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3170968448}
W20260620 19:59:35.086090 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 72875.1us
I20260620 19:59:35.080929 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2127) took 1349 ms. Trace:
I20260620 19:59:35.089412 6376 rpcz_store.cc:276] 0620 19:59:33.731870 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.145412 (+413542us) service_pool.cc:224] Handling call
0620 19:59:34.145633 (+ 221us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:34.515439 (+369806us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.080698 (+565259us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54000451d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:35.080855 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":848848128}
W20260620 19:59:34.999758 9570 consensus_meta.cc:287] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1: Time spent flushing consensus metadata: real 0.638s user 0.006s sys 0.006s
I20260620 19:59:35.096473 9648 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:34.955459 9659 raft_consensus.cc:515] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.016150 9365 leader_election.cc:290] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.053893 9578 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:34.987298 9664 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.079331 9440 leader_election.cc:290] T 1cb7a18e80e145f6913b635487750934 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.016625 9369 leader_election.cc:290] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.062259 9294 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:35.299527 6345 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:35.293836 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.228046 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1814) took 1509 ms. Trace:
W20260620 19:59:35.313238 6877 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.012s sys 0.026s
I20260620 19:59:35.318100 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:35.345010 6877 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s user 0.012s sys 0.026s
W20260620 19:59:35.346086 6301 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.018s sys 0.013s
W20260620 19:59:35.346405 6301 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s user 0.019s sys 0.013s
I20260620 19:59:35.193614 6904 raft_consensus.cc:1275] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260620 19:59:35.347394 6876 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.019s sys 0.007s
W20260620 19:59:35.347674 6876 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.019s sys 0.007s
W20260620 19:59:35.348652 6347 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.016s sys 0.012s
W20260620 19:59:35.349014 6347 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s user 0.016s sys 0.012s
I20260620 19:59:35.344646 6380 rpcz_store.cc:276] 0620 19:59:33.719151 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.024808 (+305657us) service_pool.cc:224] Handling call
0620 19:59:35.010504 (+985696us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:35.011072 (+ 568us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.227792 (+216720us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b5400065750. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.227972 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2266209024}
W20260620 19:59:35.350667 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:35.353489 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:35.292196 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:35.218029 9654 log.cc:927] Time spent T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Append to log took a long time: real 0.477s user 0.000s sys 0.001s
I20260620 19:59:35.097354 9510 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:35.345901 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.097427 9525 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.369015 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.370157 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.107833 9632 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:35.313427 6378 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.019s sys 0.019s
I20260620 19:59:35.115250 9354 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.193158 9659 leader_election.cc:290] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:35.283850 9379 raft_consensus.cc:493] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3a9a1e04ea8f425f8bcec4439ee5ebf9)
W20260620 19:59:35.365046 6906 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.010s sys 0.024s
I20260620 19:59:35.113507 9570 raft_consensus.cc:515] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.737152 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:35.737452 6378 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.100s user 0.022s sys 0.049s
I20260620 19:59:35.737596 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
W20260620 19:59:35.421029 6875 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.007s sys 0.036s
I20260620 19:59:35.864387 6902 raft_consensus.cc:1240] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: []
W20260620 19:59:35.509492 6344 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.014s sys 0.030s
W20260620 19:59:35.865082 6344 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.983s user 0.028s sys 0.043s
W20260620 19:59:35.865381 6344 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 1.06556e+06us
I20260620 19:59:35.863684 6126 catalog_manager.cc:5696] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
W20260620 19:59:35.864315 6875 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.190s user 0.008s sys 0.066s
I20260620 19:59:35.867146 6126 catalog_manager.cc:5696] T e617e76f53974a7189d97fe9f8e69a8b P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:35.417161 6304 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260620 19:59:35.671339 6196 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.001s user 0.020s sys 0.015s
W20260620 19:59:35.470109 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:35.868883 6196 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.199s user 0.020s sys 0.025s
W20260620 19:59:35.869957 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260620 19:59:35.558957 6910 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.001s sys 0.029s
W20260620 19:59:35.870460 6910 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s user 0.013s sys 0.035s
W20260620 19:59:35.737088 6906 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.079s user 0.011s sys 0.030s
W20260620 19:59:35.458405 6271 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:35.872406 9240 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:35.872037 6126 catalog_manager.cc:5696] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 8, leader changed from 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 8 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:35.872910 9240 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:35.873080 9658 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:35.743908 6197 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.976s user 0.027s sys 0.062s
I20260620 19:59:35.873781 9658 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:35.874357 6197 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.107s user 0.027s sys 0.063s
I20260620 19:59:35.874750 9236 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.875119 9592 raft_consensus.cc:493] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:35.864428 6378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1817) took 2086 ms. Trace:
I20260620 19:59:35.875809 9592 raft_consensus.cc:515] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.876911 6344 leader_election.cc:304] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:35.875017 9607 raft_consensus.cc:2749] T 1cb7a18e80e145f6913b635487750934 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:35.875288 9236 raft_consensus.cc:515] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:35.878886 6347 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:35.879544 9538 raft_consensus.cc:493] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:35.880187 9236 leader_election.cc:290] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.880085 9538 raft_consensus.cc:515] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:35.880518 6226 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 2057) took 2054 ms (client timeout 2429 ms). Trace:
I20260620 19:59:35.880889 9236 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:35.880927 6226 rpcz_store.cc:269] 0620 19:59:33.826222 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.841210 (+ 14988us) service_pool.cc:224] Handling call
0620 19:59:35.880318 (+2039108us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54001d56d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.880439 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1099070208}
I20260620 19:59:35.882000 9538 leader_election.cc:290] T ddb66400acd140a5b26f1a19f2abd18e P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.882726 9538 raft_consensus.cc:2749] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260620 19:59:35.883185 6298 raft_consensus.cc:1240] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [term 5 LEADER]: Rejecting Update request from peer 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260620 19:59:35.875872 6378 rpcz_store.cc:276] 0620 19:59:33.777631 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.214175 (+436544us) service_pool.cc:224] Handling call
0620 19:59:34.250977 (+ 36802us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:34.252486 (+ 1509us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:34.252511 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:34.252945 (+ 434us) log.cc:844] Serialized 26 byte log entry
0620 19:59:34.611111 (+358166us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:34.636964 (+ 25853us) log.cc:844] Serialized 14 byte log entry
0620 19:59:35.864089 (+1227125us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.864185 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:35.864232 (+ 47us) raft_consensus.cc:1685] finished
0620 19:59:35.864248 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:35.864373 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25531,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":3802,"raft.queue_time_us":261020,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":3491,"thread_start_us":1419643,"threads_started":2,"wal-append.queue_time_us":395712}
I20260620 19:59:35.885381 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.885703 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.886554 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.886910 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.887017 9240 leader_election.cc:290] T 93c3314daf2a488caf8f0d3a805f44cd P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.878115 9547 raft_consensus.cc:493] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.888147 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.887977 9547 raft_consensus.cc:515] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.888691 9658 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:35.562796 9375 consensus_meta.cc:287] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 0.942s user 0.010s sys 0.010s
I20260620 19:59:35.659426 9498 raft_consensus.cc:3060] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:35.750519 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 913) took 1540 ms. Trace:
I20260620 19:59:35.890131 9547 leader_election.cc:290] T 1e39bd0f2de444d2acf683f9f18716da P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.886087 6344 leader_election.cc:304] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661
I20260620 19:59:35.890458 6904 rpcz_store.cc:276] 0620 19:59:34.209884 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.932813 (+722929us) service_pool.cc:224] Handling call
0620 19:59:34.933088 (+ 275us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:35.347428 (+414340us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.750234 (+402806us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54004bd054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:35.750450 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":598269824}
I20260620 19:59:35.891741 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:35.892376 9547 raft_consensus.cc:2749] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:35.666018 9548 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:35.765250 6231 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35548 (request call id 526) took 2120 ms (client timeout 1893 ms). Trace:
W20260620 19:59:35.893464 6231 rpcz_store.cc:269] 0620 19:59:33.644959 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.645105 (+ 146us) service_pool.cc:224] Handling call
0620 19:59:35.765027 (+2119922us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54001d38d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.765164 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":592293120}
I20260620 19:59:35.768617 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 970) took 2156 ms. Trace:
I20260620 19:59:35.763710 9566 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:35.774158 6905 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53406 (request call id 890) took 3971 ms (client timeout 2937 ms). Trace:
I20260620 19:59:35.881660 9236 raft_consensus.cc:515] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.894423 6911 rpcz_store.cc:276] 0620 19:59:33.612479 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.612612 (+ 133us) service_pool.cc:224] Handling call
0620 19:59:33.695147 (+ 82535us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:33.717108 (+ 21961us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.768348 (+2051240us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b5400489750. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.768516 (+ 168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135402240}
I20260620 19:59:35.897094 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:35.897619 6911 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:35.778868 9523 consensus_queue.cc:260] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.899308 9523 raft_consensus.cc:3060] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:35.898316 9236 leader_election.cc:290] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.899842 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.782079 9464 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.785068 9475 leader_election.cc:290] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
W20260620 19:59:35.762408 9645 log.cc:927] Time spent T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.978s user 0.002s sys 0.001s
I20260620 19:59:35.810731 9559 raft_consensus.cc:493] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.903964 9559 raft_consensus.cc:515] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:35.812422 9623 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.813587 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:35.813843 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9"
I20260620 19:59:35.816215 6372 raft_consensus.cc:1275] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260620 19:59:35.817116 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 971) took 2204 ms. Trace:
I20260620 19:59:35.819366 9537 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:35.820693 9443 consensus_queue.cc:1059] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260620 19:59:35.822487 9294 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:35.834429 9445 consensus_queue.cc:1048] T 5d113f5ff7f842158ba3c35212f7bdf3 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260620 19:59:35.841102 6223 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [term 7 LEADER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260620 19:59:35.841594 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a506f4812fdc4ae2b57cb125ac69acb6" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:35.842106 6232 raft_consensus.cc:1275] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260620 19:59:35.843741 6225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c06fb2d65824956840c8faf71df4cc9" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:35.843888 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:35.898717 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:35.834748 9406 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.845181 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:35.911901 6230 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 5.
I20260620 19:59:35.906957 9623 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.849382 9518 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.852536 6380 raft_consensus.cc:1240] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:35.907253 6907 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:35.853415 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 974) took 2239 ms. Trace:
I20260620 19:59:35.914778 6908 rpcz_store.cc:276] 0620 19:59:33.613547 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.095897 (+482350us) service_pool.cc:224] Handling call
0620 19:59:34.096694 (+ 797us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:34.967338 (+870644us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.853142 (+885804us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b540052c454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:35.853340 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1982201216}
I20260620 19:59:35.916436 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:35.916977 6908 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 10.
I20260620 19:59:35.917644 6380 raft_consensus.cc:1240] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 5. Current term is 6. Ops: []
I20260620 19:59:35.907496 6377 raft_consensus.cc:3060] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Advancing to term 9
I20260620 19:59:35.907645 9549 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:35.919862 9549 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:35.921314 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:35.907743 6903 rpcz_store.cc:276] 0620 19:59:33.612757 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.612858 (+ 101us) service_pool.cc:224] Handling call
0620 19:59:33.637345 (+ 24487us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:33.637534 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 9
0620 19:59:33.637814 (+ 280us) log.cc:844] Serialized 14 byte log entry
0620 19:59:35.816554 (+2178740us) spinlock_profiling.cc:243] Waited 8.04 ms on lock 0x7b1800178d80. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6be6958f1 00007fa6be6a20c1 00007fa6b846185c 00007fa6b57abc8b 00007fa6be60a759 00007fa6b57c7e9b 00007fa6b57c613f 00007fa6b57c5c55 00007fa6b88b2440 00007fa6b88c4af6 00007fa6b88d5667
0620 19:59:35.816677 (+ 123us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:35.816741 (+ 64us) raft_consensus.cc:1645] Marking committed up to 9
0620 19:59:35.816819 (+ 78us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.816906 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:35.817047 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":131,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":49576,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":26716,"spinlock_wait_cycles":18494336,"thread_start_us":150174,"threads_started":2,"wal-append.queue_time_us":147942,"wal-append.run_cpu_time_us":1672,"wal-append.run_wall_time_us":1264215}
I20260620 19:59:35.925129 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:35.925707 6903 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:35.908696 9294 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:35.910527 9547 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:35.928382 9547 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:35.929960 6380 raft_consensus.cc:1275] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260620 19:59:35.930522 9623 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.932430 9294 leader_election.cc:290] T c5c97b15b2f74ff2a06fde66e407b188 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.910789 6225 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:35.934054 6380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93c3314daf2a488caf8f0d3a805f44cd" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:35.906483 9559 leader_election.cc:290] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:35.937683 6377 raft_consensus.cc:2468] T 52c91ae8424d495f92c57a6fcfc235d6 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:35.942734 6907 raft_consensus.cc:2468] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:35.573068 9672 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:35.868011 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 986) took 1950 ms. Trace:
I20260620 19:59:35.869868 9509 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:35.876320 9610 raft_consensus.cc:493] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.363762 9440 consensus_queue.cc:1048] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.676s
I20260620 19:59:35.878151 9592 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.881723 6196 leader_election.cc:304] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:35.887120 6302 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260620 19:59:35.878898 9388 raft_consensus.cc:3060] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:35.892845 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:35.875236 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:35.873991 6126 catalog_manager.cc:5696] T 8dba7c3eac4b4392830be760b4060cbc P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:35.777587 9342 raft_consensus.cc:3055] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Stepping down as leader of term 9
W20260620 19:59:35.895017 6905 rpcz_store.cc:269] 0620 19:59:31.803059 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:31.804118 (+ 1059us) service_pool.cc:224] Handling call
0620 19:59:35.773886 (+3969768us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b5400439ed4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.774046 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5398420224}
I20260620 19:59:35.770869 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:35.938505 9549 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:35.930347 9547 leader_election.cc:290] T 5d113f5ff7f842158ba3c35212f7bdf3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:35.908004 9537 raft_consensus.cc:515] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:35.911331 6911 raft_consensus.cc:3060] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:35.940652 6298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.941861 6304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.942298 6306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:35.594091 9667 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:35.528805 9655 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.945940 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:35.194984 9468 raft_consensus.cc:2749] T ddb66400acd140a5b26f1a19f2abd18e P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260620 19:59:35.356170 9376 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.018s sys 0.041s
I20260620 19:59:35.557520 9417 leader_election.cc:290] T 42a982b7017a41e1b0ca419a82cacf59 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725)
I20260620 19:59:35.135155 9648 raft_consensus.cc:493] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:35.147143 9578 raft_consensus.cc:2749] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260620 19:59:35.347263 9496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.007s sys 0.032s
I20260620 19:59:35.216246 9429 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:35.294360 9560 raft_consensus.cc:493] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 20bd146dc0cb4917a1a5ce05ec609ce1)
I20260620 19:59:35.219437 9603 raft_consensus.cc:493] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.947767 9445 consensus_queue.cc:1048] T c686486d28e847d0a66123f171f1149f P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:59:36.148902 6904 raft_consensus.cc:2368] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9.
I20260620 19:59:35.510084 9666 raft_consensus.cc:493] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.404641 9627 leader_election.cc:290] T a5739ffa3b914095862145adf0c084fb P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.405273 9624 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:35.418131 9507 leader_election.cc:290] T 64d372664b48432eafee6464408fcef5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:35.549268 9625 leader_election.cc:290] T dc6cbf763dbc4c3888feddf378ef4405 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:35.530050 9481 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:35.368460 9510 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:36.036522 9518 raft_consensus.cc:493] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:36.069796 9608 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.148s user 0.038s sys 0.053s
I20260620 19:59:35.667547 9569 raft_consensus.cc:493] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:36.097966 9590 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:36.428100 9665 log.cc:927] Time spent T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.864s user 0.000s sys 0.001s
I20260620 19:59:36.220214 9509 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:36.428885 9674 log.cc:927] Time spent T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.302s user 0.000s sys 0.000s
I20260620 19:59:36.440295 9466 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:36.171720 9672 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.070009 9536 consensus_queue.cc:1059] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.389s
I20260620 19:59:36.459899 9644 raft_consensus.cc:3055] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260620 19:59:36.460292 9644 raft_consensus.cc:740] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:36.460445 9672 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:36.444299 9608 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.544s user 0.050s sys 0.076s
I20260620 19:59:36.461122 9644 consensus_queue.cc:260] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.461979 9644 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:36.463939 9510 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:36.431721 9560 raft_consensus.cc:515] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.466266 9560 leader_election.cc:290] T db666bffcf904fa59d91db75dc91c8b0 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:36.335045 9570 leader_election.cc:290] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:36.445634 9669 log.cc:927] Time spent T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260620 19:59:36.448735 9460 consensus_meta.cc:287] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6: Time spent flushing consensus metadata: real 1.497s user 0.010s sys 0.001s
I20260620 19:59:36.449085 9664 leader_election.cc:290] T cc7845798261439d855e526604de104a P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:36.449903 9577 consensus_meta.cc:287] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 1.755s user 0.013s sys 0.000s
I20260620 19:59:36.450003 9590 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:36.451031 9487 raft_consensus.cc:3060] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:36.470890 9590 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:36.426265 9648 raft_consensus.cc:515] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:36.431288 9481 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:33.738652 5074 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260620 19:59:36.505931 6128 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45406 (request call id 268) took 12401 ms (client timeout 15000 ms). Trace:
W20260620 19:59:35.959394 6195 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.079s user 0.051s sys 0.135s
I20260620 19:59:36.682704 9558 raft_consensus.cc:493] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
W20260620 19:59:36.147099 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 265414us
I20260620 19:59:36.683950 9558 raft_consensus.cc:515] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.685943 9481 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:36.510653 6304 raft_consensus.cc:2410] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260620 19:59:36.687031 6379 raft_consensus.cc:2468] T 99e667a051414b87955b78b2520409b5 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
W20260620 19:59:36.511683 6381 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.865s user 0.065s sys 0.104s
I20260620 19:59:36.688515 6379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49014 (request call id 2142) took 1335 ms. Trace:
W20260620 19:59:36.688704 6381 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.042s user 0.066s sys 0.104s
I20260620 19:59:36.688879 6379 rpcz_store.cc:276] 0620 19:59:35.352608 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.353298 (+ 690us) service_pool.cc:224] Handling call
0620 19:59:36.688235 (+1334937us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b54000249d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:36.688414 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1891863808}
I20260620 19:59:36.511801 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34bcb03937cb4cea98433c0a226855de" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:35.943315 6271 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39662d01be7a44f682d160227f457661; no voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:36.690240 6229 raft_consensus.cc:2468] T 34bcb03937cb4cea98433c0a226855de P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 10.
I20260620 19:59:36.039489 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "758c2974c107466bbcc6c41d00de7b18" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.513964 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:36.127059 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.514312 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79a07a8804784405920baab70a535338" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.513440 6306 raft_consensus.cc:2468] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 11.
I20260620 19:59:36.514492 6903 raft_consensus.cc:2468] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 9.
I20260620 19:59:36.227247 6907 raft_consensus.cc:1275] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260620 19:59:35.578841 6228 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 2053) took 2517 ms (client timeout 2587 ms). Trace:
W20260620 19:59:36.516605 6272 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:36.155880 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:36.521371 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 997) took 1148 ms. Trace:
W20260620 19:59:36.170642 6345 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.005s sys 0.055s
I20260620 19:59:35.978574 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:36.015653 6874 leader_election.cc:304] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9
W20260620 19:59:36.769191 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 753392us
I20260620 19:59:36.771046 9570 raft_consensus.cc:2749] T 5ace7015fc514a909c3139c9e717b387 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 19:59:36.771206 6375 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:36.772980 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 992) took 1401 ms. Trace:
I20260620 19:59:36.773595 6903 rpcz_store.cc:276] 0620 19:59:35.371461 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.705935 (+1334474us) service_pool.cc:224] Handling call
0620 19:59:36.727634 (+ 21699us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.772537 (+ 44903us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.772576 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.772645 (+ 69us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:36.772674 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.772754 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.772892 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.529811 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.776350 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.777971 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.507306 6126 catalog_manager.cc:5696] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 7 to 8, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 8 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:36.780582 6126 catalog_manager.cc:5696] T eb61840b191946f9a272fb1b78f28cec P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 6, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 6 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:36.147928 6902 rpcz_store.cc:276] 0620 19:59:33.917580 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.356629 (+439049us) service_pool.cc:224] Handling call
0620 19:59:34.356914 (+ 285us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:35.864863 (+1507949us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.867810 (+ 2947us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54004b8cd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:35.867955 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3466119552}
I20260620 19:59:36.782874 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 991) took 1381 ms. Trace:
I20260620 19:59:36.783404 6907 rpcz_store.cc:276] 0620 19:59:35.371193 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.226024 (+854831us) service_pool.cc:224] Handling call
0620 19:59:36.226674 (+ 650us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.711743 (+485069us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.752806 (+ 41063us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.784657 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 993) took 1412 ms. Trace:
I20260620 19:59:36.785184 6902 rpcz_store.cc:276] 0620 19:59:35.371716 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.783235 (+1411519us) service_pool.cc:224] Handling call
0620 19:59:36.783690 (+ 455us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.784097 (+ 407us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.784134 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.784204 (+ 70us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:36.784246 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.784319 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.784460 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.539034 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.786665 6196 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters: 39662d01be7a44f682d160227f457661
I20260620 19:59:36.787954 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.788110 9262 consensus_queue.cc:1048] T 35398db5947a409b98b0d62a257ece06 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:36.539973 6307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59724 (request call id 798) took 1247 ms. Trace:
W20260620 19:59:36.749024 6270 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.040s sys 0.060s
I20260620 19:59:36.162883 9240 raft_consensus.cc:2804] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 19:59:36.152284 9342 raft_consensus.cc:740] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:36.097705 9354 raft_consensus.cc:493] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:36.100348 9358 consensus_meta.cc:287] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 1.727s user 0.009s sys 0.004s
W20260620 19:59:36.347407 9383 consensus_meta.cc:287] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 2.095s user 0.013s sys 0.000s
W20260620 19:59:36.424607 9376 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.770s user 0.040s sys 0.064s
I20260620 19:59:36.221607 6298 raft_consensus.cc:2368] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 8. Current term is 9.
W20260620 19:59:36.565191 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:36.853703 9577 raft_consensus.cc:2749] T 52c91ae8424d495f92c57a6fcfc235d6 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260620 19:59:36.856566 6903 raft_consensus.cc:1240] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260620 19:59:36.857952 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:36.259219 6905 raft_consensus.cc:1240] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer 39662d01be7a44f682d160227f457661 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260620 19:59:36.033977 9379 raft_consensus.cc:515] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.158456 9375 consensus_queue.cc:1048] T 9d4b5cc9261e42c381b94cd3961f1cae P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.416s
I20260620 19:59:36.162169 9236 consensus_queue.cc:1059] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.429s
W20260620 19:59:36.416818 9233 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.425s user 0.043s sys 0.136s
I20260620 19:59:36.452206 9509 leader_election.cc:290] T 0153cc2373244ab0b2574d1de595c4a4 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:36.475947 9518 raft_consensus.cc:515] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:36.452780 9523 consensus_meta.cc:287] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661: Time spent flushing consensus metadata: real 0.553s user 0.011s sys 0.000s
I20260620 19:59:36.033723 9525 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.473783 9537 leader_election.cc:290] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:36.455114 9569 raft_consensus.cc:515] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:36.912938 9233 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.886s user 0.043s sys 0.137s
I20260620 19:59:36.474763 9571 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:36.432235 9603 raft_consensus.cc:515] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.171490 9610 raft_consensus.cc:515] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.914556 9569 leader_election.cc:290] T ee78433ca73b4ea8b2b24535036f6369 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:36.476181 9648 leader_election.cc:290] T c686486d28e847d0a66123f171f1149f P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:36.426384 9666 raft_consensus.cc:515] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.920243 9625 raft_consensus.cc:493] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:36.921022 9625 raft_consensus.cc:515] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:36.921533 9642 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.006s user 0.019s sys 0.026s
W20260620 19:59:36.921983 9642 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.050s user 0.019s sys 0.027s
I20260620 19:59:36.426781 9667 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.922956 9627 raft_consensus.cc:493] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:36.923744 9627 raft_consensus.cc:515] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.925369 9518 leader_election.cc:290] T 31d307880255440b9b2c96eee7bb9f28 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:36.926098 9627 leader_election.cc:290] T a1858fc28e4948cc8e3c29e06357d166 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:36.453740 9668 log.cc:927] Time spent T 35398db5947a409b98b0d62a257ece06 P 39662d01be7a44f682d160227f457661: Append to log took a long time: real 0.326s user 0.000s sys 0.001s
I20260620 19:59:36.926914 6299 raft_consensus.cc:2410] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260620 19:59:36.927520 9367 raft_consensus.cc:493] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:36.644845 9510 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:36.928125 9367 raft_consensus.cc:515] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:36.427249 9496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.727s user 0.045s sys 0.070s
I20260620 19:59:36.033253 6225 raft_consensus.cc:2468] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 8.
I20260620 19:59:36.929188 9610 leader_election.cc:290] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:36.929801 6225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 2072) took 1087 ms. Trace:
I20260620 19:59:36.675474 6198 leader_election.cc:304] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:36.683136 6195 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.803s user 0.055s sys 0.167s
I20260620 19:59:36.516163 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 908) took 2325 ms. Trace:
I20260620 19:59:36.930917 6303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52348 (request call id 1908) took 1044 ms. Trace:
I20260620 19:59:36.691799 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d307880255440b9b2c96eee7bb9f28" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.694197 6224 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 7.
I20260620 19:59:36.690582 6379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "107df92b2ae944819f7fbc4cb216d56f" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
W20260620 19:59:36.712653 6228 rpcz_store.cc:269] 0620 19:59:33.061473 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.067217 (+ 5744us) service_pool.cc:224] Handling call
0620 19:59:35.571812 (+2504595us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54001d4cd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:35.571977 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1455611776}
I20260620 19:59:36.719080 6904 rpcz_store.cc:276] 0620 19:59:35.372778 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.892638 (+519860us) service_pool.cc:224] Handling call
0620 19:59:36.521091 (+628453us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260620 19:59:36.692248 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 748843us
I20260620 19:59:36.720772 9481 raft_consensus.cc:493] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 19:59:36.748998 6906 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.873s user 0.010s sys 0.025s
W20260620 19:59:36.749439 9386 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.008s sys 0.016s
W20260620 19:59:36.680326 6128 rpcz_store.cc:269] 0620 19:59:24.104122 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:24.104319 (+ 197us) service_pool.cc:224] Handling call
0620 19:59:24.925226 (+820907us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b54000ad054. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8824470 00007fa6b88c331a 00007fa6be678aac 00007fa6be7770ae 00007fa6bcc3c4a4 00007fa6bcc3c432 00007fa6bcc3c361 00007fa6bcc3c2dc 00007fa6bcc3b592 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:36.505523 (+11580297us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0620 19:59:28.107228 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0620 19:59:28.107784 (+ 556us) write_op.cc:432] Acquiring schema lock in shared mode
0620 19:59:28.107820 (+ 36us) write_op.cc:435] Acquired schema lock
0620 19:59:28.107842 (+ 22us) tablet.cc:662] Decoding operations
0620 19:59:28.109657 (+ 1815us) write_op.cc:620] Acquiring the partition lock for write op
0620 19:59:28.109738 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0620 19:59:28.109768 (+ 30us) tablet.cc:685] Acquiring locks for 23 operations
0620 19:59:28.453532 (+343764us) tablet.cc:701] Row locks acquired
0620 19:59:28.453555 (+ 23us) write_op.cc:260] PREPARE: finished
0620 19:59:28.453707 (+ 152us) write_op.cc:270] Start()
0620 19:59:28.453821 (+ 114us) write_op.cc:276] Timestamp: P: 1781985568453660 usec, L: 0
0620 19:59:28.453852 (+ 31us) op_driver.cc:348] REPLICATION: starting
0620 19:59:28.457073 (+ 3221us) log.cc:844] Serialized 9018 byte log entry
0620 19:59:29.070875 (+613802us) op_driver.cc:464] REPLICATION: finished
0620 19:59:29.485192 (+414317us) write_op.cc:301] APPLY: starting
0620 19:59:29.485276 (+ 84us) tablet.cc:1367] starting BulkCheckPresence
0620 19:59:29.485782 (+ 506us) tablet.cc:1370] finished BulkCheckPresence
0620 19:59:29.485903 (+ 121us) tablet.cc:1372] starting ApplyRowOperation cycle
0620 19:59:29.690579 (+204676us) tablet.cc:1383] finished ApplyRowOperation cycle
0620 19:59:29.690981 (+ 402us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=23
0620 19:59:29.691045 (+ 64us) write_op.cc:312] APPLY: finished
0620 19:59:29.691610 (+ 565us) log.cc:844] Serialized 158 byte log entry
0620 19:59:29.691801 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:29.692049 (+ 248us) write_op.cc:454] Released schema lock
0620 19:59:29.692149 (+ 100us) write_op.cc:341] FINISH: Updating metrics
0620 19:59:29.692657 (+ 508us) write_op.cc:489] Releasing partition, row and schema locks
0620 19:59:29.692701 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":407147,"reported_tablets":37,"spinlock_wait_cycles":9743616,"child_traces":[["op",{"apply.queue_time_us":410601,"apply.run_cpu_time_us":6828,"apply.run_wall_time_us":207418,"num_ops":23,"prepare.queue_time_us":222629,"prepare.run_cpu_time_us":32681,"prepare.run_wall_time_us":553942,"raft.queue_time_us":233461,"raft.run_cpu_time_us":26771,"raft.run_wall_time_us":308736,"replication_time_us":589941,"thread_start_us":575577,"threads_started":4,"wal-append.queue_time_us":257134,"wal-append.run_cpu_time_us":22723,"wal-append.run_wall_time_us":2008485}]]}
I20260620 19:59:36.755954 6302 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
W20260620 19:59:36.756670 6345 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.253s user 0.005s sys 0.055s
I20260620 19:59:36.763144 6911 raft_consensus.cc:2468] T 758c2974c107466bbcc6c41d00de7b18 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:36.766636 9558 leader_election.cc:290] T 6d08b9c2da1448139f24e1d94c9616f3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:36.794732 6270 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.040s sys 0.060s
I20260620 19:59:36.773401 6375 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []
I20260620 19:59:36.789624 6307 rpcz_store.cc:276] 0620 19:59:35.292063 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.292962 (+ 899us) service_pool.cc:224] Handling call
0620 19:59:35.293315 (+ 353us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.514686 (+1221371us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.514766 (+ 80us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.514848 (+ 82us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:36.514945 (+ 97us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.539379 (+ 24434us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54003a6454. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:36.539427 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.539661 (+ 234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2807068416}
I20260620 19:59:36.801277 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56f959cee394e398810b9170b9d24be" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.931396 6910 rpcz_store.cc:276] 0620 19:59:34.190713 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.927900 (+737187us) service_pool.cc:224] Handling call
0620 19:59:34.928303 (+ 403us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:34.928650 (+ 347us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:34.928676 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:34.929093 (+ 417us) log.cc:844] Serialized 26 byte log entry
0620 19:59:35.870843 (+941750us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:35.870880 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.870944 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:36.515469 (+644525us) raft_consensus.cc:1685] finished
0620 19:59:36.515518 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.515810 (+ 292us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7578,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":1320,"thread_start_us":918778,"threads_started":1,"wal-append.queue_time_us":1504660}
I20260620 19:59:36.939294 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35398db5947a409b98b0d62a257ece06" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.940418 9379 leader_election.cc:290] T cc7845798261439d855e526604de104a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:36.819463 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:36.827103 6273 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.028s sys 0.021s
I20260620 19:59:36.940696 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 918) took 1046 ms. Trace:
W20260620 19:59:36.941051 6273 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.789s user 0.029s sys 0.021s
W20260620 19:59:36.941514 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 789738us
I20260620 19:59:36.941752 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cb7a18e80e145f6913b635487750934" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.942914 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 919) took 1048 ms. Trace:
I20260620 19:59:36.943231 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a982b7017a41e1b0ca419a82cacf59" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.943427 6904 rpcz_store.cc:276] 0620 19:59:35.894230 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.941662 (+1047432us) service_pool.cc:224] Handling call
0620 19:59:36.942123 (+ 461us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.942510 (+ 387us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.942540 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.942624 (+ 84us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:36.942666 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.942729 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.942855 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.944000 6227 raft_consensus.cc:2468] T 42a982b7017a41e1b0ca419a82cacf59 P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
I20260620 19:59:36.944536 9603 leader_election.cc:290] T 65e57f1c5f9f437db061394645beec68 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:36.945211 6227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.945845 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 920) took 1051 ms. Trace:
W20260620 19:59:36.949427 9386 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.107s user 0.009s sys 0.017s
W20260620 19:59:36.828985 6908 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.010s sys 0.025s
I20260620 19:59:36.835915 6126 catalog_manager.cc:5696] T a56f959cee394e398810b9170b9d24be P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 9 to 11, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 11 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260620 19:59:36.950907 6908 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s user 0.010s sys 0.026s
W20260620 19:59:36.837203 9408 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.978s user 0.025s sys 0.025s
W20260620 19:59:36.837679 6380 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.005s sys 0.034s
I20260620 19:59:36.844473 9644 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260620 19:59:36.937493 6906 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.066s user 0.010s sys 0.026s
I20260620 19:59:36.848826 9354 raft_consensus.cc:515] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.850122 9358 raft_consensus.cc:2749] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725) with higher term. Message: Invalid argument: T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for earlier term 8. Current term is 9.
I20260620 19:59:36.851151 9383 raft_consensus.cc:515] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:36.829490 9240 raft_consensus.cc:697] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [term 5 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:36.878839 9687 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260620 19:59:36.881259 9570 raft_consensus.cc:493] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:36.794541 6877 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.391s user 0.019s sys 0.041s
W20260620 19:59:37.071321 6877 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.725s user 0.020s sys 0.045s
W20260620 19:59:37.071784 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 1.72572e+06us
I20260620 19:59:36.890923 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 990) took 1504 ms. Trace:
I20260620 19:59:37.073036 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 988) took 2842 ms. Trace:
I20260620 19:59:37.073540 6908 rpcz_store.cc:276] 0620 19:59:34.230247 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.944167 (+1713920us) service_pool.cc:224] Handling call
0620 19:59:35.944682 (+ 515us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:35.945160 (+ 478us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:35.945219 (+ 59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:36.033454 (+ 88235us) log.cc:844] Serialized 26 byte log entry
0620 19:59:37.072341 (+1038887us) spinlock_profiling.cc:243] Waited 62.8 ms on lock 0x7b54004dd120. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8824470 00007fa6b8831a98 00007fa6b88bad0b 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c
0620 19:59:37.072443 (+ 102us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:37.072512 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.072599 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:37.072652 (+ 53us) raft_consensus.cc:1685] finished
0620 19:59:37.072666 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.072965 (+ 299us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":315,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":111,"spinlock_wait_cycles":144332160,"thread_start_us":917611,"threads_started":1,"wal-append.queue_time_us":913343}
I20260620 19:59:37.072664 6905 rpcz_store.cc:276] 0620 19:59:35.370909 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.154233 (+783324us) service_pool.cc:224] Handling call
0620 19:59:36.258895 (+104662us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:36.874519 (+615624us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.874934 (+ 415us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.079376 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 921) took 1184 ms. Trace:
I20260620 19:59:36.890935 6196 leader_election.cc:304] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9
I20260620 19:59:36.895049 9631 raft_consensus.cc:493] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:36.914036 9571 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.072662 6877 leader_election.cc:304] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:37.082612 6300 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 10.
I20260620 19:59:37.082748 9496 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.083340 9496 raft_consensus.cc:515] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.084224 9571 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:36.919425 6305 raft_consensus.cc:1240] T 84b845bdbf9e462d9997ac5cf41386c3 P 39662d01be7a44f682d160227f457661 [term 10 LEADER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []
W20260620 19:59:37.085039 6300 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 793) took 3388 ms (client timeout 1771 ms). Trace:
I20260620 19:59:37.085556 9496 leader_election.cc:290] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:37.085443 6300 rpcz_store.cc:269] 0620 19:59:33.697037 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:33.697178 (+ 141us) service_pool.cc:224] Handling call
0620 19:59:37.084784 (+3387606us) spinlock_profiling.cc:243] Waited 3.23 s on lock 0x7b54001a9d54. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:37.084927 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7438843648}
I20260620 19:59:37.086194 9496 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.086336 6305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1772) took 2635 ms. Trace:
I20260620 19:59:37.087073 9544 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.087365 9687 raft_consensus.cc:2804] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 FOLLOWER]: Leader election won for term 10
I20260620 19:59:37.088007 9687 raft_consensus.cc:697] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 10 LEADER]: Becoming Leader. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:37.087743 9544 raft_consensus.cc:515] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.089282 9687 consensus_queue.cc:237] T 92d22c99dfc64cc1bb999795e691d1ad P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.090029 9544 leader_election.cc:290] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:36.921893 9666 leader_election.cc:290] T 43e3501e8f31428cb18618fc6a8e1f59 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.090903 9666 raft_consensus.cc:493] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:36.928310 9510 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.091310 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.705492 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:36.457443 9498 consensus_meta.cc:287] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.514s user 0.005s sys 0.002s
I20260620 19:59:36.693387 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.093422 9510 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:37.093541 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9f062c40bd4cb68d470ec9b5e63a92" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.093953 9548 raft_consensus.cc:493] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.086776 6305 rpcz_store.cc:276] 0620 19:59:34.450710 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.452475 (+ 1765us) service_pool.cc:224] Handling call
0620 19:59:34.522375 (+ 69900us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.084951 (+2562576us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.086056 (+ 1105us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b54001a9354. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:37.086277 (+ 221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5502428032}
I20260620 19:59:37.094329 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:36.931738 9679 raft_consensus.cc:2804] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260620 19:59:37.094767 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64d372664b48432eafee6464408fcef5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.094724 9548 raft_consensus.cc:515] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:36.931124 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 256094us
I20260620 19:59:37.096029 6228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52c91ae8424d495f92c57a6fcfc235d6" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.955719 9367 leader_election.cc:290] T 2e56dc56fbc548ac997024a1b2e2ea1a P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.101974 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6"
I20260620 19:59:36.957731 6299 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 805) took 1220 ms. Trace:
I20260620 19:59:37.105085 6299 rpcz_store.cc:276] 0620 19:59:35.737201 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.737358 (+ 157us) service_pool.cc:224] Handling call
0620 19:59:36.957433 (+1220075us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003255d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:36.957627 (+ 194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2442633984}
I20260620 19:59:36.959592 6374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1828) took 1109 ms. Trace:
I20260620 19:59:37.107968 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d113f5ff7f842158ba3c35212f7bdf3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.108037 6374 rpcz_store.cc:276] 0620 19:59:35.849630 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.849765 (+ 135us) service_pool.cc:224] Handling call
0620 19:59:35.850034 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.922998 (+1072964us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.923051 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.923138 (+ 87us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:36.923186 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.959147 (+ 35961us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000ec254. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:36.959187 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.959411 (+ 224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2466457856}
W20260620 19:59:37.109413 9487 consensus_meta.cc:287] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9: Time spent flushing consensus metadata: real 0.638s user 0.003s sys 0.018s
I20260620 19:59:36.954969 6226 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260620 19:59:37.110322 6373 raft_consensus.cc:1275] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260620 19:59:37.111625 6373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2131) took 3263 ms. Trace:
I20260620 19:59:36.957430 6126 catalog_manager.cc:5696] T 8cf0b0e2dafa4512b82ecefda3013d79 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 5, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 5 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:37.111928 6373 rpcz_store.cc:276] 0620 19:59:33.848416 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.263846 (+415430us) service_pool.cc:224] Handling call
0620 19:59:34.264237 (+ 391us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.110723 (+2846486us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.111385 (+ 662us) spinlock_profiling.cc:243] Waited 2.85 s on lock 0x7b54002cb0d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:37.111581 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6543611776}
I20260620 19:59:37.113394 9673 raft_consensus.cc:493] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.113991 9673 raft_consensus.cc:515] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.113336 6126 catalog_manager.cc:5696] T 31d307880255440b9b2c96eee7bb9f28 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:37.114944 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 922) took 1220 ms. Trace:
I20260620 19:59:37.115844 6126 catalog_manager.cc:5696] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 4, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:37.116933 9673 leader_election.cc:290] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.118362 9548 leader_election.cc:290] T af9232a7f82e4c3f911b856592bd6ba9 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.118299 9496 raft_consensus.cc:515] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.120307 6875 leader_election.cc:304] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:37.121323 9569 raft_consensus.cc:2749] T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260620 19:59:37.121910 9496 leader_election.cc:290] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:37.126395 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.126932 9460 raft_consensus.cc:493] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.127270 6226 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:37.127614 9460 raft_consensus.cc:515] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.079648 6273 leader_election.cc:304] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:36.933301 6303 rpcz_store.cc:276] 0620 19:59:35.884736 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.884870 (+ 134us) service_pool.cc:224] Handling call
0620 19:59:35.885156 (+ 286us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.912962 (+1027806us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.913018 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.913063 (+ 45us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:36.913114 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.928772 (+ 15658us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400325ad4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:36.928816 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.928983 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2356433792}
I20260620 19:59:36.941154 6910 rpcz_store.cc:276] 0620 19:59:35.894004 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.939604 (+1045600us) service_pool.cc:224] Handling call
0620 19:59:36.939965 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.940340 (+ 375us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.940375 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.940440 (+ 65us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:36.940481 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.940542 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.940642 (+ 100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.946264 6904 rpcz_store.cc:276] 0620 19:59:35.894473 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.944708 (+1050235us) service_pool.cc:224] Handling call
0620 19:59:36.945103 (+ 395us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.945427 (+ 324us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.945455 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.945514 (+ 59us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:36.945582 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.945652 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.945768 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.914559 9525 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.137544 9525 raft_consensus.cc:2804] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:37.137903 9525 raft_consensus.cc:493] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.138252 9525 raft_consensus.cc:3060] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:37.138177 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.138911 6224 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 10.
I20260620 19:59:36.844930 9342 consensus_queue.cc:260] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.142055 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e667a051414b87955b78b2520409b5" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.968809 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53408 (request call id 998) took 1591 ms. Trace:
W20260620 19:59:36.972828 6380 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.000s user 0.006s sys 0.035s
I20260620 19:59:37.142575 6907 rpcz_store.cc:276] 0620 19:59:35.373033 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.784396 (+1411363us) service_pool.cc:224] Handling call
0620 19:59:36.784847 (+ 451us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.923207 (+138360us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.923243 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.923297 (+ 54us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:36.923351 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.963825 (+ 40474us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54000b95d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:36.963862 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.963975 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":299780480}
I20260620 19:59:37.143321 6380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2145) took 1173 ms. Trace:
I20260620 19:59:36.975476 9570 raft_consensus.cc:515] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.143908 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.145232 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.145619 9570 leader_election.cc:290] T e617e76f53974a7189d97fe9f8e69a8b P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:37.145848 6907 raft_consensus.cc:2468] T 8b0538f91fad4439a7d30adbf2f83b57 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 9.
I20260620 19:59:36.976207 9518 raft_consensus.cc:493] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:36.940631 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e56dc56fbc548ac997024a1b2e2ea1a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.146734 9518 raft_consensus.cc:515] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.147403 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43e3501e8f31428cb18618fc6a8e1f59" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.148468 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.148792 9518 leader_election.cc:290] T cfeb1fb4ea6746dbaf0509f4097e00ab P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.148983 6231 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
W20260620 19:59:36.972534 9408 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.150s user 0.026s sys 0.026s
I20260620 19:59:37.149780 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0538f91fad4439a7d30adbf2f83b57" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.977159 6376 raft_consensus.cc:1275] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Refusing update from remote peer 39662d01be7a44f682d160227f457661: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260620 19:59:36.973917 6302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59724 (request call id 806) took 1025 ms. Trace:
I20260620 19:59:37.150732 6302 rpcz_store.cc:276] 0620 19:59:35.948771 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.155577 (+206806us) service_pool.cc:224] Handling call
0620 19:59:36.970547 (+814970us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.978101 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 989) took 1602 ms. Trace:
I20260620 19:59:37.151650 6224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.985555 6906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 907) took 2795 ms. Trace:
I20260620 19:59:37.152366 6224 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 8.
I20260620 19:59:37.152333 6906 rpcz_store.cc:276] 0620 19:59:34.190369 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.591844 (+401475us) service_pool.cc:224] Handling call
0620 19:59:34.635361 (+ 43517us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:34.635560 (+ 199us) raft_consensus.cc:1534] Early marking committed up to index 10
0620 19:59:34.657537 (+ 21977us) log.cc:844] Serialized 14 byte log entry
0620 19:59:36.982798 (+2325261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.985052 (+ 2254us) raft_consensus.cc:1645] Marking committed up to 10
0620 19:59:36.985169 (+ 117us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:36.985272 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.985476 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32657,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":11210,"raft.queue_time_us":18540,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"thread_start_us":2308696,"threads_started":2,"wal-append.queue_time_us":936899}
I20260620 19:59:36.986580 6909 raft_consensus.cc:2393] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39662d01be7a44f682d160227f457661 in current term 5: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:37.155223 6231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:37.155558 6909 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53408 (request call id 996) took 1783 ms (client timeout 1547 ms). Trace:
W20260620 19:59:37.155933 6909 rpcz_store.cc:269] 0620 19:59:35.372529 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.770528 (+397999us) service_pool.cc:224] Handling call
0620 19:59:37.155347 (+1384819us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b54006e0354. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:37.155495 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1912225024}
I20260620 19:59:37.157145 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.158224 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.000077 9354 leader_election.cc:290] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.001935 9383 leader_election.cc:290] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 9 election: Requested vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.003005 9240 consensus_queue.cc:237] T 758c2974c107466bbcc6c41d00de7b18 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.011257 6903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 916) took 1117 ms. Trace:
I20260620 19:59:37.177376 6903 rpcz_store.cc:276] 0620 19:59:35.893500 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.898334 (+1004834us) service_pool.cc:224] Handling call
0620 19:59:36.934225 (+ 35891us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.010925 (+ 76700us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:37.010953 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.010996 (+ 43us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:37.011034 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.011089 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.011190 (+ 101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.980348 9233 consensus_queue.cc:1048] T a00974f881fb476fb0715a861ffcf358 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260620 19:59:37.020916 6227 raft_consensus.cc:1275] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Refusing update from remote peer 20bd146dc0cb4917a1a5ce05ec609ce1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260620 19:59:37.026062 9358 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.027056 9608 raft_consensus.cc:3055] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Stepping down as leader of term 8
I20260620 19:59:37.026938 9558 raft_consensus.cc:493] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.027841 9603 raft_consensus.cc:493] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.192023 9603 raft_consensus.cc:515] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.028577 9610 raft_consensus.cc:493] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.194809 9358 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.029464 9627 raft_consensus.cc:493] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.034787 6381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46186 (request call id 640) took 3143 ms. Trace:
I20260620 19:59:37.195892 9627 raft_consensus.cc:515] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.197640 9358 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.198051 9627 leader_election.cc:290] T eb61840b191946f9a272fb1b78f28cec P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.196043 6381 rpcz_store.cc:276] 0620 19:59:33.889345 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.346506 (+457161us) service_pool.cc:224] Handling call
0620 19:59:34.392848 (+ 46342us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:34.409891 (+ 17043us) raft_consensus.cc:1534] Early marking committed up to index 7
0620 19:59:34.515474 (+105583us) log.cc:844] Serialized 14 byte log entry
0620 19:59:35.217477 (+702003us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:36.689265 (+1471788us) log.cc:844] Serialized 14 byte log entry
0620 19:59:36.910241 (+220976us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.916228 (+ 5987us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:36.932881 (+ 16653us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.020657 (+ 87776us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b54001207d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:37.020880 (+ 223us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.032809 (+ 11929us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":130504,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":3658,"raft.queue_time_us":7743,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":1306,"spinlock_wait_cycles":43046912,"thread_start_us":2446682,"threads_started":4,"wal-append.queue_time_us":421856,"wal-append.run_cpu_time_us":13919,"wal-append.run_wall_time_us":1705871}
I20260620 19:59:37.039784 9625 leader_election.cc:290] T 5ace7015fc514a909c3139c9e717b387 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.058557 6372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49028 (request call id 1831) took 1143 ms. Trace:
I20260620 19:59:37.199065 6372 rpcz_store.cc:276] 0620 19:59:35.910048 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.910202 (+ 154us) service_pool.cc:224] Handling call
0620 19:59:35.910441 (+ 239us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:36.976437 (+1065996us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:36.976491 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:36.976536 (+ 45us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:36.976569 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.051122 (+ 74553us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54000235d4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:37.051161 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.053545 (+ 2384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2310537472}
I20260620 19:59:37.079700 6905 rpcz_store.cc:276] 0620 19:59:35.894723 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.078198 (+1183475us) service_pool.cc:224] Handling call
0620 19:59:37.078683 (+ 485us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.079027 (+ 344us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:37.079048 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.079109 (+ 61us) raft_consensus.cc:1645] Marking committed up to 8
0620 19:59:37.079135 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.079200 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.079313 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.091461 6375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.092473 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5739ffa3b914095862145adf0c084fb" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.095314 9679 raft_consensus.cc:697] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [term 9 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:36.909649 9683 raft_consensus.cc:493] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.096143 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:36.930532 6301 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52350 (request call id 1773) took 2479 ms. Trace:
I20260620 19:59:36.930258 6225 rpcz_store.cc:276] 0620 19:59:35.842001 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.843521 (+ 1520us) service_pool.cc:224] Handling call
0620 19:59:36.929573 (+1086052us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.099414 9631 raft_consensus.cc:515] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:37.100111 6196 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 210998us
I20260620 19:59:37.201326 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72968f173fdb4542920e97a940db9668" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.205993 6372 raft_consensus.cc:2468] T 72968f173fdb4542920e97a940db9668 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 6.
I20260620 19:59:37.091405 9666 raft_consensus.cc:515] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.207239 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "098d4ada753442e1899d391a485095aa" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.207846 6372 raft_consensus.cc:2468] T 098d4ada753442e1899d391a485095aa P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 9.
I20260620 19:59:37.208211 9666 leader_election.cc:290] T 48323077f2994fe383c12116b6e89657 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.208871 6372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00974f881fb476fb0715a861ffcf358" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.209183 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e39bd0f2de444d2acf683f9f18716da" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.115334 6908 rpcz_store.cc:276] 0620 19:59:35.894946 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.078820 (+1183874us) service_pool.cc:224] Handling call
0620 19:59:37.079282 (+ 462us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.114615 (+ 35333us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:37.114639 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.114680 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
0620 19:59:37.114742 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.114799 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.114894 (+ 95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.210271 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1858fc28e4948cc8e3c29e06357d166" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.210745 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb66400acd140a5b26f1a19f2abd18e" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.211001 6908 raft_consensus.cc:2468] T a1858fc28e4948cc8e3c29e06357d166 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 8.
I20260620 19:59:37.211932 6905 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.212025 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dba7c3eac4b4392830be760b4060cbc" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.213100 6905 raft_consensus.cc:2410] T 97eee360f90e4475858b88d88f8efc5c P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260620 19:59:37.213027 6908 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:37.135582 9460 leader_election.cc:290] T 8b96addaf2b149a2ba7decbb292167e5 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
I20260620 19:59:37.187968 9608 raft_consensus.cc:740] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:37.214883 6375 raft_consensus.cc:2468] T 8dba7c3eac4b4392830be760b4060cbc P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:37.100816 9375 raft_consensus.cc:2749] T a5739ffa3b914095862145adf0c084fb P eb2f228ebe40440d877be6302bba2cd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260620 19:59:37.136698 6273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 138653us
I20260620 19:59:37.151577 6911 rpcz_store.cc:276] 0620 19:59:35.370534 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.898346 (+527812us) service_pool.cc:224] Handling call
0620 19:59:36.968241 (+1069895us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:36.968380 9644 raft_consensus.cc:493] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:37.218477 9644 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Advancing to term 11
I20260620 19:59:36.984153 9481 raft_consensus.cc:515] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.214875 9683 raft_consensus.cc:515] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.224292 9481 leader_election.cc:290] T 9d4b5cc9261e42c381b94cd3961f1cae P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:37.153409 6378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.225052 6378 raft_consensus.cc:2468] T 03268245eb53439a9b88defe52240125 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 6.
I20260620 19:59:37.225103 9683 leader_election.cc:290] T 779819d94546427fa367040185e417ab P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.226128 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:37.227142 9679 consensus_queue.cc:237] T 52c91ae8424d495f92c57a6fcfc235d6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:36.987856 9667 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:36.986768 6379 raft_consensus.cc:2468] T 107df92b2ae944819f7fbc4cb216d56f P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 5.
I20260620 19:59:37.158757 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e57f1c5f9f437db061394645beec68" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:37.204027 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 62455.9us
I20260620 19:59:37.158912 6902 raft_consensus.cc:1240] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 4. Current term is 5. Ops: []
I20260620 19:59:37.159842 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.161634 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
W20260620 19:59:37.160383 9537 consensus_queue.cc:1175] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" responder_term: 10 status { last_received { term: 9 index: 5 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260620 19:59:37.163064 6376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49014 (request call id 2143) took 1810 ms. Trace:
I20260620 19:59:37.154291 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc6cbf763dbc4c3888feddf378ef4405" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:37.143648 6380 rpcz_store.cc:276] 0620 19:59:35.970266 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.971187 (+ 921us) service_pool.cc:224] Handling call
0620 19:59:35.971554 (+ 367us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:35.971834 (+ 280us) raft_consensus.cc:1534] Early marking committed up to index 7
0620 19:59:35.972117 (+ 283us) log.cc:844] Serialized 14 byte log entry
0620 19:59:37.142965 (+1170848us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.143043 (+ 78us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:37.143104 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.143182 (+ 78us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.143276 (+ 94us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":785373,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":3296,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"thread_start_us":1170429,"threads_started":1,"wal-append.queue_time_us":922615}
I20260620 19:59:37.176671 6223 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:37.179036 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.191040 9558 raft_consensus.cc:515] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.194957 9610 raft_consensus.cc:515] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.043071 6271 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.202941 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae1222c89dd47e78f503964b6a95973" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.194361 9603 leader_election.cc:290] T 84b845bdbf9e462d9997ac5cf41386c3 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:37.209332 9675 raft_consensus.cc:493] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.142158 9342 raft_consensus.cc:3060] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:37.208284 6903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ace7015fc514a909c3139c9e717b387" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.204274 6301 rpcz_store.cc:276] 0620 19:59:34.450975 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:34.451229 (+ 254us) service_pool.cc:224] Handling call
0620 19:59:34.451517 (+ 288us) raft_consensus.cc:1471] Updating replica for 1 ops
0620 19:59:34.523318 (+ 71801us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:34.523362 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0620 19:59:34.523845 (+ 483us) log.cc:844] Serialized 26 byte log entry
0620 19:59:35.346840 (+822995us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:35.346884 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:35.346971 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0620 19:59:36.453218 (+1106247us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b440002250c. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b5651a01 00007fa6b56ef485 00007fa6b56efea2 00007fa6b846557b 00007fa6b8460e4f 00007fa6b8454c0e 00007fa6b8453f56 00007fa6b8453b78 00007fa6b845d5c1 00007fa6b845d579 00007fa6b845d509
0620 19:59:36.929894 (+476676us) raft_consensus.cc:1685] finished
0620 19:59:36.929941 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:36.930274 (+ 333us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79428,"raft.queue_time_us":75296,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":64489,"spinlock_wait_cycles":52828544,"thread_start_us":822736,"threads_started":1,"wal-append.queue_time_us":1152424}
I20260620 19:59:37.243798 6907 raft_consensus.cc:2468] T 65e57f1c5f9f437db061394645beec68 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 7.
I20260620 19:59:37.217303 9608 consensus_queue.cc:260] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.244257 6906 raft_consensus.cc:3060] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:37.218053 6911 raft_consensus.cc:1275] T 2e56dc56fbc548ac997024a1b2e2ea1a P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260620 19:59:37.227576 6307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5c97b15b2f74ff2a06fde66e407b188" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.129998 6875 leader_election.cc:304] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1; no voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:37.234625 6908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 923) took 1322 ms. Trace:
I20260620 19:59:37.245136 6908 rpcz_store.cc:276] 0620 19:59:35.895195 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.213778 (+1318583us) service_pool.cc:224] Handling call
0620 19:59:37.214196 (+ 418us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.214511 (+ 315us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:37.214542 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.214582 (+ 40us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:37.216931 (+ 2349us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.216993 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.217104 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3968}
I20260620 19:59:37.245051 6911 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 925) took 1349 ms. Trace:
I20260620 19:59:37.209546 6372 raft_consensus.cc:2468] T a00974f881fb476fb0715a861ffcf358 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 10.
W20260620 19:59:37.238533 6197 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:37.246230 6911 rpcz_store.cc:276] 0620 19:59:35.895646 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.217335 (+1321689us) service_pool.cc:224] Handling call
0620 19:59:37.217675 (+ 340us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.244826 (+ 27151us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.245002 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.231410 6376 rpcz_store.cc:276] 0620 19:59:35.352862 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:35.352986 (+ 124us) service_pool.cc:224] Handling call
0620 19:59:35.353262 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.150407 (+1797145us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.162798 (+ 12391us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b540028dcd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88bb1e1 00007fa6b88b9c1e 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076
0620 19:59:37.163003 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3732893440}
W20260620 19:59:37.242463 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 219498us
I20260620 19:59:37.242899 6905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 924) took 1340 ms. Trace:
I20260620 19:59:37.238564 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b96addaf2b149a2ba7decbb292167e5" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.247612 6905 rpcz_store.cc:276] 0620 19:59:35.895419 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.213780 (+1318361us) service_pool.cc:224] Handling call
0620 19:59:37.214132 (+ 352us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.214528 (+ 396us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:37.214557 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.217094 (+ 2537us) raft_consensus.cc:1645] Marking committed up to 9
0620 19:59:37.235967 (+ 18873us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.236010 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.236081 (+ 71us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.250057 9354 consensus_queue.cc:1048] T 03268245eb53439a9b88defe52240125 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:59:37.250409 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cda425863f04d209a457c07ec3064d1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661"
I20260620 19:59:37.250684 9408 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 19:59:37.251127 9408 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.244856 6377 raft_consensus.cc:2468] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 7.
W20260620 19:59:37.251432 6875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 125418us
I20260620 19:59:37.250185 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97eee360f90e4475858b88d88f8efc5c" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.250180 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cf0b0e2dafa4512b82ecefda3013d79" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.252802 6302 raft_consensus.cc:2410] T 8cf0b0e2dafa4512b82ecefda3013d79 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260620 19:59:37.251255 6299 raft_consensus.cc:2393] T 6cda425863f04d209a457c07ec3064d1 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 9: Already voted for candidate eb2f228ebe40440d877be6302bba2cd6 in this term.
I20260620 19:59:37.242216 6126 catalog_manager.cc:5696] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 3 to 4, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 4 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260620 19:59:37.254479 6305 raft_consensus.cc:2393] T 97eee360f90e4475858b88d88f8efc5c P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in current term 9: Already voted for candidate 39662d01be7a44f682d160227f457661 in this term.
I20260620 19:59:37.257258 6909 raft_consensus.cc:2393] T eae1222c89dd47e78f503964b6a95973 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in current term 7: Already voted for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in this term.
I20260620 19:59:37.256309 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.261122 9667 raft_consensus.cc:2749] T 31d307880255440b9b2c96eee7bb9f28 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 19:59:37.263489 6904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 928) took 1367 ms. Trace:
I20260620 19:59:37.263674 9610 leader_election.cc:290] T f202638d921b4ef2b75223ca54d70e04 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.254642 9408 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 4 FOLLOWER]: Advancing to term 5
I20260620 19:59:37.264221 9558 leader_election.cc:290] T aacb05a304e442a7bd73148aff5f4a66 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.264743 6300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d08b9c2da1448139f24e1d94c9616f3" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.266425 9608 raft_consensus.cc:3060] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Advancing to term 9
I20260620 19:59:37.263877 6904 rpcz_store.cc:276] 0620 19:59:35.896313 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.251869 (+1355556us) service_pool.cc:224] Handling call
0620 19:59:37.262955 (+ 11086us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.263188 (+ 233us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:37.263213 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.263250 (+ 37us) raft_consensus.cc:1645] Marking committed up to 11
0620 19:59:37.263293 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.263340 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.263439 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.268700 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af9232a7f82e4c3f911b856592bd6ba9" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.269251 6904 raft_consensus.cc:2468] T af9232a7f82e4c3f911b856592bd6ba9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 5.
I20260620 19:59:37.269986 6875 leader_election.cc:304] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.270573 6910 raft_consensus.cc:1275] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260620 19:59:37.270977 6910 pending_rounds.cc:85] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting all ops after (but not including) 8
I20260620 19:59:37.271220 6910 pending_rounds.cc:107] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260620 19:59:37.271461 6910 raft_consensus.cc:2889] T 79a07a8804784405920baab70a535338 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260620 19:59:37.272161 6910 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 926) took 1376 ms. Trace:
I20260620 19:59:37.272445 9570 raft_consensus.cc:2804] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:37.272863 9570 raft_consensus.cc:493] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:37.273083 6908 raft_consensus.cc:1240] T a506f4812fdc4ae2b57cb125ac69acb6 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Rejecting Update request from peer eb2f228ebe40440d877be6302bba2cd6 for earlier term 6. Current term is 7. Ops: []
I20260620 19:59:37.273213 9570 raft_consensus.cc:3060] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:37.274055 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03268245eb53439a9b88defe52240125" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.270011 9687 consensus_queue.cc:1059] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.005s
I20260620 19:59:37.275646 9687 consensus_queue.cc:1048] T 0153cc2373244ab0b2574d1de595c4a4 P 20bd146dc0cb4917a1a5ce05ec609ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 19:59:37.276856 6902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 917) took 1383 ms. Trace:
I20260620 19:59:37.277258 9687 raft_consensus.cc:3055] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Stepping down as leader of term 7
I20260620 19:59:37.277750 9687 raft_consensus.cc:740] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 20bd146dc0cb4917a1a5ce05ec609ce1, State: Running, Role: LEADER
I20260620 19:59:37.277782 6907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53396 (request call id 927) took 1381 ms. Trace:
I20260620 19:59:37.279449 6271 leader_election.cc:304] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.278380 6907 rpcz_store.cc:276] 0620 19:59:35.896078 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.252307 (+1356229us) service_pool.cc:224] Handling call
0620 19:59:37.277320 (+ 25013us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.277480 (+ 160us) raft_consensus.cc:1534] Early marking committed up to index 0
0620 19:59:37.277503 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0620 19:59:37.277556 (+ 53us) raft_consensus.cc:1645] Marking committed up to 7
0620 19:59:37.277600 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.277650 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0620 19:59:37.277735 (+ 85us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.279282 9687 consensus_queue.cc:260] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.282795 9233 raft_consensus.cc:2804] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260620 19:59:37.282953 9687 raft_consensus.cc:3060] T 1c06fb2d65824956840c8faf71df4cc9 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:37.283236 9233 raft_consensus.cc:493] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:37.283607 9233 raft_consensus.cc:3060] T 34bcb03937cb4cea98433c0a226855de P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Advancing to term 11
I20260620 19:59:37.283706 9631 leader_election.cc:290] T 7a566dbaea0746f4bc76e04b5072e045 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.261999 9644 raft_consensus.cc:515] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.272455 6910 rpcz_store.cc:276] 0620 19:59:35.895860 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:37.251755 (+1355895us) service_pool.cc:224] Handling call
0620 19:59:37.270180 (+ 18425us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.271938 (+ 1758us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.272104 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260620 19:59:37.288322 9644 leader_election.cc:290] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 election: Requested vote from peers 39662d01be7a44f682d160227f457661 (127.4.244.130:42141), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:37.288926 6875 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:37.291352 6905 raft_consensus.cc:1275] T cfeb1fb4ea6746dbaf0509f4097e00ab P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 7 FOLLOWER]: Refusing update from remote peer 3a9a1e04ea8f425f8bcec4439ee5ebf9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260620 19:59:37.292236 9675 raft_consensus.cc:515] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.294448 9642 consensus_queue.cc:1048] T 2e56dc56fbc548ac997024a1b2e2ea1a P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:37.290987 9408 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.297809 9537 consensus_queue.cc:1059] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:59:37.298435 9408 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 5 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
I20260620 19:59:37.298826 9537 raft_consensus.cc:3055] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:37.299253 9537 raft_consensus.cc:740] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:37.280900 6271 leader_election.cc:304] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 39662d01be7a44f682d160227f457661, 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.299458 6198 leader_election.cc:304] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.277218 6902 rpcz_store.cc:276] 0620 19:59:35.893776 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.925925 (+1032149us) service_pool.cc:224] Handling call
0620 19:59:37.158703 (+232778us) raft_consensus.cc:1471] Updating replica for 0 ops
0620 19:59:37.234858 (+ 76155us) raft_consensus.cc:1697] Filling consensus response to leader.
0620 19:59:37.276662 (+ 41804us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54006e0350. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88b9c8d 00007fa6bd87eb19 00007fa6b6df3d34 00007fa6b6df3cc2 00007fa6b6df3bf1 00007fa6b6df3b6c 00007fa6b6df2e22 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:37.276775 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":534233472}
I20260620 19:59:37.290300 6906 raft_consensus.cc:2468] T dc6cbf763dbc4c3888feddf378ef4405 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 6.
I20260620 19:59:37.310204 6910 raft_consensus.cc:1275] T 6cda425863f04d209a457c07ec3064d1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Refusing update from remote peer eb2f228ebe40440d877be6302bba2cd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260620 19:59:37.310544 6195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 58057.8us
I20260620 19:59:37.316766 6198 leader_election.cc:304] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.318367 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7845798261439d855e526604de104a" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.315431 6126 catalog_manager.cc:5696] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 7, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 7 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:37.294538 9675 leader_election.cc:290] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701), 20bd146dc0cb4917a1a5ce05ec609ce1 (127.4.244.132:42325)
W20260620 19:59:37.316939 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 147025us
I20260620 19:59:37.331195 6906 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1"
I20260620 19:59:37.333030 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.333568 6271 leader_election.cc:304] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39662d01be7a44f682d160227f457661, eb2f228ebe40440d877be6302bba2cd6; no voters: 20bd146dc0cb4917a1a5ce05ec609ce1
I20260620 19:59:37.337050 6271 leader_election.cc:304] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:37.338213 6305 raft_consensus.cc:1240] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Rejecting Update request from peer 20bd146dc0cb4917a1a5ce05ec609ce1 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260620 19:59:37.319993 9544 raft_consensus.cc:2804] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:37.339397 9544 raft_consensus.cc:493] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:37.339756 9544 raft_consensus.cc:3060] T 098d4ada753442e1899d391a485095aa P eb2f228ebe40440d877be6302bba2cd6 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:37.319826 9496 raft_consensus.cc:2804] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:37.343034 6906 raft_consensus.cc:3060] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Advancing to term 11
I20260620 19:59:37.325915 9354 raft_consensus.cc:2804] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260620 19:59:37.341270 6126 catalog_manager.cc:5696] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 2 to 7, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 7 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:37.346800 9367 raft_consensus.cc:2804] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 8 FOLLOWER]: Leader election won for term 8
I20260620 19:59:37.358232 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.358232 6305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.362224 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee78433ca73b4ea8b2b24535036f6369" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.344626 9496 raft_consensus.cc:493] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.349956 9354 raft_consensus.cc:493] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:37.362126 6230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45808 (request call id 2076) took 1323 ms. Trace:
I20260620 19:59:37.370110 6271 leader_election.cc:304] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 39662d01be7a44f682d160227f457661; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:37.366206 9642 consensus_queue.cc:1048] T 79a07a8804784405920baab70a535338 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260620 19:59:37.374737 9560 raft_consensus.cc:2804] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Leader election won for term 7
I20260620 19:59:37.356644 6126 catalog_manager.cc:5696] T 99e667a051414b87955b78b2520409b5 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 4 to 6, leader changed from 39662d01be7a44f682d160227f457661 (127.4.244.130) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 6 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:37.353276 9537 consensus_queue.cc:260] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.375831 6910 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.376288 9367 raft_consensus.cc:697] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [term 8 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:37.377749 6198 leader_election.cc:304] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.391274 9570 raft_consensus.cc:515] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
W20260620 19:59:37.401356 6270 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:37.402057 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.393671 6230 rpcz_store.cc:276] 0620 19:59:36.039111 (+ 0us) service_pool.cc:167] Inserting onto call queue
0620 19:59:36.039236 (+ 125us) service_pool.cc:224] Handling call
0620 19:59:37.353405 (+1314169us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54001d4cd4. stack: 00007fa6b5796b43 00007fa6b5796a40 00007fa6b5124574 00007fa6bee696c4 00007fa6bee69659 00007fa6b8823951 00007fa6b88be6da 00007fa6bd880217 00007fa6b6df8834 00007fa6b6df87c2 00007fa6b6df86f1 00007fa6b6df866c 00007fa6b6df7922 00007fa6b6bf2c3c 00007fa6b6bf2076 00007fa6b6bf1a1f
0620 19:59:37.361980 (+ 8575us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1450259200}
I20260620 19:59:37.417279 6223 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 10.
I20260620 19:59:37.410110 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48323077f2994fe383c12116b6e89657" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.418200 6906 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 11.
I20260620 19:59:37.410836 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb61840b191946f9a272fb1b78f28cec" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.411473 9518 raft_consensus.cc:2804] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260620 19:59:37.412055 6226 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84b845bdbf9e462d9997ac5cf41386c3" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.427204 6226 raft_consensus.cc:2468] T 84b845bdbf9e462d9997ac5cf41386c3 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 10.
I20260620 19:59:37.412945 6305 raft_consensus.cc:2410] T 48323077f2994fe383c12116b6e89657 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20bd146dc0cb4917a1a5ce05ec609ce1 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260620 19:59:37.413738 9367 consensus_queue.cc:237] T 1c06fb2d65824956840c8faf71df4cc9 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.410769 6299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5c21aa29d76d48fc933f76007685dae1" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.418375 6908 raft_consensus.cc:2468] T 48323077f2994fe383c12116b6e89657 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a9a1e04ea8f425f8bcec4439ee5ebf9 in term 9.
I20260620 19:59:37.420143 9570 leader_election.cc:290] T 8504518fd53942459d685f3abcbae2eb P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 6 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 39662d01be7a44f682d160227f457661 (127.4.244.130:42141)
I20260620 19:59:37.420689 6302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8504518fd53942459d685f3abcbae2eb" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.421366 6303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e617e76f53974a7189d97fe9f8e69a8b" candidate_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "39662d01be7a44f682d160227f457661" is_pre_election: true
I20260620 19:59:37.429375 6299 raft_consensus.cc:2468] T 5c21aa29d76d48fc933f76007685dae1 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 10.
I20260620 19:59:37.430547 6302 raft_consensus.cc:2468] T 8504518fd53942459d685f3abcbae2eb P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:37.421944 6877 leader_election.cc:304] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: eb2f228ebe40440d877be6302bba2cd6
I20260620 19:59:37.393293 9496 raft_consensus.cc:3060] T 72968f173fdb4542920e97a940db9668 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
W20260620 19:59:37.425097 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 61555.6us
I20260620 19:59:37.394768 9354 raft_consensus.cc:3060] T 42a982b7017a41e1b0ca419a82cacf59 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Advancing to term 8
I20260620 19:59:37.426049 6196 leader_election.cc:304] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, eb2f228ebe40440d877be6302bba2cd6; no voters:
W20260620 19:59:37.442423 6196 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:37.442546 6126 catalog_manager.cc:5696] T 43e3501e8f31428cb18618fc6a8e1f59 P eb2f228ebe40440d877be6302bba2cd6 reported cstate change: term changed from 1 to 7, leader changed from 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131) to eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129). New cstate: current_term: 7 leader_uuid: "eb2f228ebe40440d877be6302bba2cd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260620 19:59:37.431212 6303 raft_consensus.cc:2468] T e617e76f53974a7189d97fe9f8e69a8b P 39662d01be7a44f682d160227f457661 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bd146dc0cb4917a1a5ce05ec609ce1 in term 5.
I20260620 19:59:37.450021 6198 leader_election.cc:304] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a1e04ea8f425f8bcec4439ee5ebf9, eb2f228ebe40440d877be6302bba2cd6; no voters:
I20260620 19:59:37.431193 9675 raft_consensus.cc:2804] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260620 19:59:37.449528 6232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:37.603811 6271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.132:42325, user_credentials={real_user=slave}} blocked reactor thread for 233761us
I20260620 19:59:37.428755 9518 raft_consensus.cc:493] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
W20260620 19:59:37.622347 6345 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.129:36725, user_credentials={real_user=slave}} blocked reactor thread for 45403.9us
I20260620 19:59:37.604023 6230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
I20260620 19:59:37.426362 9342 raft_consensus.cc:493] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:59:37.633477 6223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:37.630028 6198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 185569us
I20260620 19:59:37.432273 9560 raft_consensus.cc:697] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [term 7 LEADER]: Becoming Leader. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
I20260620 19:59:37.505877 9537 raft_consensus.cc:3060] T a506f4812fdc4ae2b57cb125ac69acb6 P eb2f228ebe40440d877be6302bba2cd6 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:37.524240 9569 raft_consensus.cc:2804] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:37.603598 9525 raft_consensus.cc:515] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.487241 9631 raft_consensus.cc:493] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.588168 9388 raft_consensus.cc:493] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.438349 9262 consensus_queue.cc:1048] T 6cda425863f04d209a457c07ec3064d1 P eb2f228ebe40440d877be6302bba2cd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.202s
I20260620 19:59:37.538810 9677 raft_consensus.cc:2804] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 11 FOLLOWER]: Leader election won for term 11
I20260620 19:59:37.594466 9460 raft_consensus.cc:493] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.606930 9558 raft_consensus.cc:493] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.625612 9675 raft_consensus.cc:493] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 39662d01be7a44f682d160227f457661)
I20260620 19:59:37.629839 9518 raft_consensus.cc:3060] T 79a07a8804784405920baab70a535338 P 39662d01be7a44f682d160227f457661 [term 9 FOLLOWER]: Advancing to term 10
I20260620 19:59:37.572337 6229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "eb2f228ebe40440d877be6302bba2cd6" is_pre_election: true
W20260620 19:59:37.655975 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.131:41701, user_credentials={real_user=slave}} blocked reactor thread for 234082us
W20260620 19:59:37.657150 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 216633us
I20260620 19:59:37.651661 6374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92d22c99dfc64cc1bb999795e691d1ad" candidate_uuid: "eb2f228ebe40440d877be6302bba2cd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.638080 9342 raft_consensus.cc:515] T 0153cc2373244ab0b2574d1de595c4a4 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.639618 9375 raft_consensus.cc:2804] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260620 19:59:37.644219 9569 raft_consensus.cc:493] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.685400 9388 raft_consensus.cc:515] T 92212144534e4394844003016ba67a47 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.685983 9569 raft_consensus.cc:3060] T 8dba7c3eac4b4392830be760b4060cbc P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Advancing to term 7
I20260620 19:59:37.686954 9375 raft_consensus.cc:493] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:59:37.687374 9375 raft_consensus.cc:3060] T a00974f881fb476fb0715a861ffcf358 P eb2f228ebe40440d877be6302bba2cd6 [term 10 FOLLOWER]: Advancing to term 11
I20260620 19:59:37.688002 6377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d4b5cc9261e42c381b94cd3961f1cae" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.679690 6374 raft_consensus.cc:2468] T 92d22c99dfc64cc1bb999795e691d1ad P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f228ebe40440d877be6302bba2cd6 in term 10.
I20260620 19:59:37.680397 6908 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779819d94546427fa367040185e417ab" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.692569 6909 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.696455 6911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aacb05a304e442a7bd73148aff5f4a66" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.696700 6907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f202638d921b4ef2b75223ca54d70e04" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.698403 6904 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a566dbaea0746f4bc76e04b5072e045" candidate_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" is_pre_election: true
I20260620 19:59:37.676975 9675 raft_consensus.cc:3060] T 107df92b2ae944819f7fbc4cb216d56f P eb2f228ebe40440d877be6302bba2cd6 [term 5 FOLLOWER]: Advancing to term 6
I20260620 19:59:37.688151 6373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db666bffcf904fa59d91db75dc91c8b0" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.650106 9608 consensus_queue.cc:1059] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260620 19:59:37.701599 6877 leader_election.cc:304] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters: 39662d01be7a44f682d160227f457661
W20260620 19:59:37.705327 6273 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260620 19:59:37.705714 9408 raft_consensus.cc:493] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader eb2f228ebe40440d877be6302bba2cd6)
I20260620 19:59:37.710927 9560 consensus_queue.cc:237] T a506f4812fdc4ae2b57cb125ac69acb6 P 39662d01be7a44f682d160227f457661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.731768 6373 raft_consensus.cc:2468] T db666bffcf904fa59d91db75dc91c8b0 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39662d01be7a44f682d160227f457661 in term 6.
I20260620 19:59:37.722218 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c686486d28e847d0a66123f171f1149f" candidate_uuid: "39662d01be7a44f682d160227f457661" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" is_pre_election: true
I20260620 19:59:37.729123 9691 consensus_queue.cc:1048] T af9232a7f82e4c3f911b856592bd6ba9 P 39662d01be7a44f682d160227f457661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260620 19:59:37.691897 9460 raft_consensus.cc:515] T 64d372664b48432eafee6464408fcef5 P eb2f228ebe40440d877be6302bba2cd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
W20260620 19:59:37.758237 6874 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.244.130:42141, user_credentials={real_user=slave}} blocked reactor thread for 56872.1us
I20260620 19:59:37.729770 9525 leader_election.cc:290] T 99e667a051414b87955b78b2520409b5 P 39662d01be7a44f682d160227f457661 [CANDIDATE]: Term 7 election: Requested vote from peers eb2f228ebe40440d877be6302bba2cd6 (127.4.244.129:36725), 3a9a1e04ea8f425f8bcec4439ee5ebf9 (127.4.244.131:41701)
W20260620 19:59:37.745887 9707 log.cc:927] Time spent T 35398db5947a409b98b0d62a257ece06 P 20bd146dc0cb4917a1a5ce05ec609ce1: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260620 19:59:37.742185 9608 raft_consensus.cc:3055] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Stepping down as leader of term 6
I20260620 19:59:37.711874 9558 raft_consensus.cc:515] T c5c97b15b2f74ff2a06fde66e407b188 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } } peers { permanent_uuid: "20bd146dc0cb4917a1a5ce05ec609ce1" member_type: VOTER last_known_addr { host: "127.4.244.132" port: 42325 } attrs { promote: false } }
I20260620 19:59:37.700966 9631 raft_consensus.cc:515] T 93c3314daf2a488caf8f0d3a805f44cd P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.753780 9687 raft_consensus.cc:2804] T 03268245eb53439a9b88defe52240125 P 20bd146dc0cb4917a1a5ce05ec609ce1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260620 19:59:37.733291 9408 raft_consensus.cc:515] T 5d113f5ff7f842158ba3c35212f7bdf3 P 39662d01be7a44f682d160227f457661 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2f228ebe40440d877be6302bba2cd6" member_type: VOTER last_known_addr { host: "127.4.244.129" port: 36725 } } peers { permanent_uuid: "39662d01be7a44f682d160227f457661" member_type: VOTER last_known_addr { host: "127.4.244.130" port: 42141 } } peers { permanent_uuid: "3a9a1e04ea8f425f8bcec4439ee5ebf9" member_type: VOTER last_known_addr { host: "127.4.244.131" port: 41701 } }
I20260620 19:59:37.781394 6345 leader_election.cc:304] T 8b0538f91fad4439a7d30adbf2f83b57 P 3a9a1e04ea8f425f8bcec4439ee5ebf9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bd146dc0cb4917a1a5ce05ec609ce1, 3a9a1e04ea8f425f8bcec4439ee5ebf9; no voters:
I20260620 19:59:37.786301 9677 raft_consensus.cc:697] T 5c21aa29d76d48fc933f76007685dae1 P eb2f228ebe40440d877be6302bba2cd6 [term 11 LEADER]: Becoming Leader. State: Replica: eb2f228ebe40440d877be6302bba2cd6, State: Running, Role: LEADER
I20260620 19:59:37.776522 9608 raft_consensus.cc:740] T eae1222c89dd47e78f503964b6a95973 P 39662d01be7a44f682d160227f457661 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 39662d01be7a44f682d160227f457661, State: Running, Role: LEADER
W20260620 19:59:37.807402 6877 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote ->